Must have Skills: Apache Spark (Strong), Azure Data Factory, Azure Table storage,
Good To Have Skills: Databricks,
Job Summary:
We are seeking a skilled Data Engineer with a minimum of 8+ years experience. The ideal candidate will have a strong background in batch data processing, developing and maintaining scalable ETL/ELT pipelines, and working with cloud platforms like Azure.
This role focuses on building efficient data solutions that ensure high performance and data quality.
Key Responsibilities:
- Design, develop, and maintain scalable batch ETL/ELT pipelines using Azure Data Factory and other Azure services.
- Collaborate with data architects and cross-functional teams to translate business requirements into technical solutions.
- Ensure data processing pipelines are optimized for performance, reliability, and cost-effectiveness.
- Perform data integration, cleansing, and validation to ensure high data quality.
- Monitor and troubleshoot batch processing jobs to ensure timely and accurate data delivery.
- Ensure compliance with data governance and security standards.
Qualifications:
- Minimum of 8+ years of experience in Data Engineering, focusing on batch data processing.
- Hands-on experience with Azure data services (Azure Data Factory, Data Lake, etc.).
- Strong understanding of ETL processes, data integration, and data modeling.
- Experience with data storage solutions such as Azure Data Lake and SQL databases.
- Strong problem-solving and communication skills.
