Key Responsibilities:
Architecture & Platform Design
- Design enterprise Databricks Lakehouse architectures aligned with the Databricks Well-Architected Framework
- Define reference architectures for batch, streaming, analytics, and ML workloads
- Select and standardize cluster, compute, and workspace architectures
- Design multi-workspace strategies (dev/test/prod, shared vs. isolated)
- Ensure architectures meet scalability, availability, and performance requirements
Well-Architected Framework Alignment
Apply Databricks best practices across all pillars, including:
-
Security & Governance (Unity Catalog, IAM, data access controls)
-
Reliability & Resilience (job retries, checkpointing, failure isolation)
-
Performance Efficiency (cluster sizing, autoscaling, caching)
-
Cost Optimization (compute policies, workload separation, monitoring)
-
Operational Excellence (monitoring, automation, CI/CD, runbooks)
Implementation & Engineering
- Lead Databricks workspace, cluster, and Unity Catalog implementations
- Implement Delta Lake, Delta Live Tables (DLT), and Structured Streaming
- Build and optimize ETL/ELT pipelines using Spark and SQL
- Integrate Databricks with cloud services (S3/ADLS/GCS, IAM, Key Vault, networking)
- Establish CI/CD pipelines for notebooks, jobs, and infrastructure
Security, Governance & Compliance
- Implement Unity Catalog for centralized governance
- Define data classification, lineage, and audit strategies
- Enforce least-privilege access and secure networking patterns
- Support compliance requirements (HIPAA, SOC 2, PCI, GDPR as applicable)
Operations & Optimization
- Monitor platform health, performance, and cost
- Troubleshoot production issues across jobs, clusters, and data pipelines
- Perform workload tuning and cost-performance optimization
- Define SLOs, alerts, and operational metrics
Collaboration & Leadership
- Partner with Data Engineering, Analytics, ML, Platform, and Security teams
- Translate business requirements into technical architectures
- Provide architectural guidance and technical mentorship
- Communicate risks, tradeoffs, and recommendations to leadership
Required Qualifications:
Experience
-
7+ years in data engineering, analytics, or platform architecture
-
3–5+ years hands-on Databricks experience in production environments
- Proven experience applying the Databricks Well-Architected Framework
- Experience designing cloud-native lakehouse architectures
- Experience supporting mission-critical data platforms
Technical Skills
- Databricks Lakehouse Platform
- Apache Spark (PySpark / Scala / Spark SQL)
- Delta Lake, Delta Live Tables, Structured Streaming
- Unity Catalog (governance, lineage, access controls)
- Cloud platforms: AWS, Azure, or GCP
- Infrastructure as Code (Terraform strongly preferred)
- CI/CD tools (GitHub Actions, Azure DevOps, GitLab, etc.)
- Data formats and protocols (Parquet, JSON, Avro)
Certifications Required:
- Databricks Certified Data Engineer Professional
-
Databricks Certified Professional Architect (or equivalent advanced certification)
Preferred / Additional Certifications
- AWS Certified Solutions Architect (Associate or Professional)
- Azure Solutions Architect Expert
- Google Professional Data Engineer
- Databricks Machine Learning Professional
- Snowflake or other cloud data platform certifications
Soft Skills
- Strong architectural decision-making and documentation skills
- Excellent communication with technical and non-technical stakeholders
- Ability to lead design reviews and architecture governance forums
- Strong troubleshooting and performance-tuning mindset
Nice-to-Have Experience
- MLflow and MLOps architectures
- Real-time analytics and streaming pipelines
- Multi-region or cross-account data architectures
- Consulting or MSP delivery experience
