Senior Staff Engineer - Cloud Architect

TLDR

Design and manage cloud data platforms on Azure with a focus on data governance, security, and optimization using Databricks and Microsoft's Well-Architected Framework.

Cloud Architect for Azure Databricks designs and manages data platforms on Azure, leveraging Databricks for unified data science, engineering, and analytics, focusing on the two-plane architecture (Control & Data Planes), data governance (Unity Catalog), security (Entra ID, Key Vault), cost optimization, and building reliable Lakehouse solutions using Microsoft's Well-Architected Framework principles for data pipelines (Bronze, Silver, Gold layers) and integration with other Azure services like Synapse, Power BI, and Azure Monitor.
Architecture Design: Designing end-to-end data solutions, from ingestion to consumption, using Databricks'''' Lakehouse Platform on Azure.
Control Plane vs. Data Plane: Understanding the separation where Databricks manages the backend (Control Plane) while the customer owns the compute and data (Data Plane) within their Azure subscription, ensuring data residency and security.
Data Governance: Implementing Unity Catalog for centralized access control, lineage, and security across data assets.
Security & Identity: Integrating with Microsoft Entra ID (Azure AD) for SSO, using Azure Key Vault for secrets, and Microsoft Defender for Cloud for protection.
Performance & Cost: Optimizing cluster configurations (classic vs. serverless), storage (ADLS Gen2), and resource usage for cost efficiency.

Must have Skills: Cloud development (Strong), Cloud architecture (Strong), Databricks