As an Azure Databricks Engineer, you will play a pivotal role in our technology solutions team, focusing on the deployment and management of Azure Databricks environments. This position requires a deep understanding of cloud-based data engineering and analytics, particularly within the Azure ecosystem. You will be responsible for implementing best practices in CI/CD deployment using Databricks Asset Bundles (DAB) and Azure DevOps, ensuring that our data solutions are robust, scalable, and efficient. Your expertise will help drive our clients' digital transformation initiatives by leveraging the power of data analytics and machine learning.
Job Responsibilities
- Collaborate with cross-functional teams to design, implement, and manage Azure Databricks environments.
- Deploy Azure Databricks and configure workspaces to meet project requirements.
- Create and manage Databricks Asset Bundles (DAB) to facilitate CI/CD deployment processes.
- Implement CI/CD pipelines using Azure DevOps, ensuring seamless integration and delivery of data solutions.
- Manage Unity Catalog changes and workspace configurations to enhance data governance and accessibility.
- Utilize Terraform for infrastructure as code (IaC) to automate the provisioning of Databricks resources.
- Monitor and optimize Databricks performance, ensuring efficient resource utilization and cost management.
- Provide technical guidance and support to team members and stakeholders regarding best practices in data engineering and analytics.
- Stay updated with the latest trends and advancements in Azure Databricks and related technologies to continuously improve our solutions.
Requirements
- Proven experience in configuring Azure Databricks Workspaces.
- Expertise in creating Databricks Asset Bundles (DAB) for CI/CD deployment.
- Strong experience with Azure DevOps, including GIT and Pipelines.
- Demonstrated experience as an Azure Databricks Workspace Administrator.
- Proficient in managing Databricks Unity Catalog for data governance.
- Familiarity with Terraform for infrastructure automation is a plus.
- Solid understanding of data engineering principles and practices.
- Excellent problem-solving skills and the ability to work collaboratively in a team environment.
- Strong communication skills to effectively convey technical concepts to non-technical stakeholders.
