Senior Azure Databricks Engineer

AHEAD builds platforms for digital business. By weaving together advances in cloud infrastructure, automation and analytics, and software delivery, we help enterprises deliver on the promise of digital transformation. At AHEAD, we prioritize creating a culture of belonging, where all perspectives and voices are represented, valued, respected, and heard. We create spaces to empower everyone to speak up, make change, and drive the culture at AHEAD.  We are an equal opportunity employer, and do not discriminate based on an individual's race, national origin, color, gender, gender identity, gender expression, sexual orientation, religion, age, disability, marital status, or any other protected characteristic under applicable law, whether actual or perceived.  We embrace all candidates that will contribute to the diversification and enrichment of ideas and perspectives at AHEAD.  What you’ll do

Architect, build, and optimize Azure Databricks solutions (batch + streaming) using Apache Spark. 

Design Lakehouse data models and curation layers using Delta Lake (Bronze/Silver/Gold), including SCD patterns where applicable. 

Lead end-to-end development of ETL/ELT pipelines integrating sources like APIs, RDBMS, SaaS apps, files, and event streams. 

Implement orchestration and scheduling using Databricks Workflows and/or Azure Data Factory (or Airflow where relevant). 

Drive performance and cost optimization: cluster policies, autoscaling, photon/runtime selection, partitioning, file sizing, job tuning. 

Establish best practices for code quality: modular PySpark/SQL, reusable libraries, unit/integration testing, and peer reviews. 

Implement CI/CD for Databricks (Repos/Git), automated deployments, environment promotion, and release governance. 

Partner with security/governance teams to implement access controls, secrets, encryption, and data governance (e.g., Unity Catalog). 

Create operational excellence: monitoring, alerting, runbooks, incident response, and root-cause analysis. 

Mentor engineers, set technical direction, and contribute to roadmap planning and estimation. 

Required qualifications

6+ years of data engineering experience, including strong hands-on delivery with Azure Databricks. 

Expert-level Spark experience (PySpark and/or Scala) and strong SQL. 

Proven experience building production-grade Delta Lake / Lakehouse architectures and data pipelines. 

Strong experience with Azure services such as ADLS Gen2, Azure Data Factory, Event Hubs/Kafka, Synapse, Key Vault, and Azure networking/security concepts. 

Demonstrated ability to tune Spark workloads (joins/shuffles, partitioning, caching, skew handling, file compaction). 

Experience with Git-based development and CI/CD (Azure DevOps or GitHub Actions), plus automated testing approaches. 

Strong communication skills and ability to lead technical discussions, influence decisions, and translate requirements into designs. 

Preferred qualifications

Databricks certifications (Data Engineer Associate/Professional). 

Experience with Unity Catalog, catalog/lineage, role-based access control, and enterprise governance. 

Infrastructure as Code (Terraform/Bicep) and platform automation experience. 

Experience with data quality/observability tools (e.g., Great Expectations, Deequ, Monte Carlo—any equivalent). 

Familiarity with ML enablement on Databricks (MLflow, feature engineering, MLOps) and/or BI integration (Power BI). 

Key competencies

Systems thinking and pragmatic architecture decisions 

Ownership mindset, reliability focus, and production operations maturity 

Mentorship and technical leadership (standards, patterns, reviews) 

Azure Databricks, Spark (PySpark/Scala), SQL, Delta Lake, ADLS Gen2, ADF, Event Hubs/Kafka, Key Vault, Unity Catalog, Azure DevOps/GitHub, Terraform