Senior Lead Data Engineer - R01557349

AI overview

Design and implement ELT pipelines and optimize infrastructure for enterprise data warehouses while mentoring team members in best data engineering practices.

Senior Lead Data Engineer Primary Skills

Architect, construct, test and maintain ELT pipelines using Hadoop and Spark SQL

Specialization

Hadoop and Spark SQL

Job requirements

Key Responsibilities:

Architect, construct , test and maintain ELT pipelines using Hadoop and Spark SQL

- Manage and optimize the infrastructure ensuring reliability and performance

-Design and implement data models and schemas for enterprise data warehouse and data marts

-Write and maintain scripts in Python for various applications

-Develop, optimize, troubleshoot complex SQL queries and database structures for relational and non-relational databases

- Work closely with analysts and product managers to understand data requirements and deliver high-quality data

-Provide technical guidance and mentorship to other team members fostering best practices in data engineering

-Monitor system performance, identify bottlenecks, implement solutions to ensure high data availability

Required Skills:

-High proficiency in the Hadoop ecosystem (HDFS, MapReduce, Yarn) and Apache Spark, Scala for large scale data processing

-Good knowledge in Python for data manipulation and scripting

-Mastery of advanced SQL and deep experience working with relational databases and datawarehouses

-Hands-on experience with workflow management tools like Apache Airflow

-Familiarity with Unix shell scripting for automation and system management