We are looking for a highly skilled Senior Data Engineer to join our team. Your responsibilities:
- Contribute to the technical planning and execution of data migration projects, focusing on data ingestion, transformation, storage, and access control within Azure Data Factory and Data Lake
- Design and develop scalable data pipelines using Azure Databricks to streamline data movement from diverse data sources
- Build reusable frameworks for efficient data ingestion and transformation.
- Implement robust mechanisms for data validation, cleansing, and integrity checks to ensure high-quality data across pipelines
- Work with event-based/streaming technologies to ingest and process real-time data
- Provide technical support and resolve challenges during migration and post-migration phases
- Collaborate with internal teams to troubleshoot and optimize existing data solutions
- Stay updated with the latest cloud, data engineering, and analytics technologies.
- Recommend and implement industry best practices for scalable and efficient data solutions
-
5+ years of overall IT experience with a focus on data engineering and hands-on experience with Azure Databricks
- Strong expertise in Python programming, particularly for data engineering and automation
- Experience with PySpark and proficiency in SQL for data modeling and source system analysis
- Extensive experience using Azure services such as: Azure Data Factory, Azure Data Lake, Azure SQL Data Warehouse, Azure SQL
- Proven ability to conduct data profiling, cataloging, and mapping to design and implement technical data flows
- Familiarity with data visualization tools like Power BI for data exploration and reporting
- Experience in preparing data for Data Science and Machine Learning projects is advantageous
- Proficiency with Jupyter Notebooks or Databricks Notebooks for Python development (preferred)
- Knowledge of event-based streaming technologies (e.g., Kafka, Event Hubs) is a plus
