We are seeking a highly skilled and experienced Engineering Manager to lead and manage our big data and cloud-based solutions team. The ideal candidate will have deep expertise in AWS services like EMR, Athena, Glue, S3, and Apache Airflow, along with strong proficiency in Big Data Technologies using Python and PySpark.
As an Engineering Manager, you will be responsible for designing, developing, and optimizing large-scale Batch data processing applications, while also leading a team of talented engineers. You will work closely with cross-functional teams to deliver scalable and reliable solutions that enable high-performance data processing and analytics in the cloud.
Key Responsibilities:
- Lead, mentor, and manage a team of engineers working on cloud-based big data applications and services.
- Design, develop, and maintain large-scale Batch data processing pipelines using AWS services (EMR, Athena, Glue, S3) and Apache Airflow.
- Architect and implement data solutions using Python, PySpark, and big data frameworks to handle complex datasets efficiently.
- Collaborate with product management, Business, Platform Infrastructure and DevOps teams to define project requirements and deliver high-quality, scalable solutions.
- Ensure the team's development efforts follow best practices in coding, security, and data management.
- Optimize performance, reliability, and cost-effectiveness of data pipelines and cloud infrastructure.
- Drive continuous improvement in engineering processes, ensuring the team’s deliverables meet quality and performance benchmarks.
- Stay updated on emerging technologies and industry trends, and guide the adoption of new tools and processes to enhance engineering efficiency.
-
Bachelor’s degree in Computer Science, Engineering, or a related field.
-
8 - 10 years of experience in software engineering, with at least 3+ years in a leadership or management role.
- Strong hands-on experience with AWS services including EMR, Athena, Glue, S3, and Apache Airflow.
- Expertise in Big Data technologies, especially with Python, PySpark, and distributed data processing frameworks.
- Proven experience in designing and developing scalable and reliable data pipelines and applications.
- Solid understanding of data architecture, data modeling, and ETL processes.
- Strong leadership, communication, and project management skills, with the ability to collaborate effectively with cross-functional teams.
- A background in cloud infrastructure and DevOps practices for big data environments is a plus.
- Experience with other cloud platforms or tools, such as Google Cloud Platform (GCP) or Azure.
- Knowledge of data warehousing, data lakes, and streaming data technologies.
- Experience working in an Agile/Scrum development environment.
Experian Careers - Creating a better tomorrow together
Find out what its like to work for Experian by clicking here
