At 3Pillar, our focus is on leveraging cutting-edge technologies that revolutionize industries by enabling data driven decision making. As a Lead, Data Engineering, you will hold a crucial position within our dynamic team, actively contributing to thrilling projects that reshape data analytics for our clients, providing them with a competitive advantage in their respective industries.
Key Responsibilities
- Understanding the client requirement and architect robust data platforms on multiple cloud technologies.
- Take the lead on creating, defining and driving the data engineering solutions for complex client engagements
- Develop ETL/ELT processes to help extract and manipulate data from multiple sources.
- Work with the leadership to set the standards for software engineering practices within the data engineering team
- Maintain and optimize the data infrastructure required for accurate extraction, transformation, and loading of data from a wide variety of data sources.
- Monitor data systems performance and implement optimization strategies.
- Ensure data accuracy, integrity, privacy, security, and compliance through quality control procedures.
- Partner with data scientists and functional leaders in sales, marketing, and product to deploy machine learning models in production.
- Choose and use the right analytical libraries, programming languages, and frameworks for each task.
- Lead data engineering team.
Minimum Qualifications
- Demonstrated expertise with a minimum of 7+ years of experience in data engineering domain
- Advanced SQL skills and experience with relational databases and database design.
- Experience with data modeling
- Experience working with cloud Data Warehouse solutions (e.g., Snowflake, Redshift, BigQuery, Azure Synapse, etc.).
- Strong Python skills with hands-on experience on Pandas, NumPy and other data related libraries
- Experience with working on large data sets and distributed computing (e.g. Hive/Hadoop/Spark/Presto/MapReduce).
- In depth knowledge and understanding of data streaming tools like Apache Kafka, AWS Kinesis.
- Proficient in data pipeline and workflow management tools e.g. Airflow, AWS Step functions etc
- Experience with data visualization tools like PowerBI, Tableau, AWs Quicksight etc
- Experience with the AWS/Azure/GCP data engineering services
- Knowledge of AWS services viz., EC2, EMR, S3, Lambda, EBS, IAM, Redshift, RDS, GLUE ETL
- OR
- Knowledge of Azure services viz., ADF, Databricks, Azure Synapse Analytics, ADLS Gen2, Azure Devops
- Good understanding of NoSQL databases like Redis, Cassandra, MongoDB, or Neo4j.
- Excellent problem-solving, communication, and organizational skills.
- Proven ability to manage a highly skilled data engineering team.
