The Position
We're looking for a Data Engineer to join our amazing Data team. Your main tasks will include working with different stakeholders to improve our data platform's performance and quality. You'll also design and set up reliable pipelines to manage data from various sources. Plus, you'll help keep our Airflow system running smoothly. We are looking for someone to ensure our data meets high standards for quality, security, and speed.
Responsibilities
- Collaborate with stakeholders to define, develop, and implement enhancements to our data platform, prioritizing performance and quality at scale
- Design and implement resilient Extract, Load, Transform (ELT) pipelines from a variety of data sources and tools
- Maintain and optimize our Airflow infrastructure to ensure efficient orchestration processes
- Champion engineering best practices, including documentation, code review, and automated testing in a Continuous Integration/Continuous Deployment (CI/CD) environment
- Advocate for improvements in data quality, security, and performance
You have
- 3+ years of relevant experience in data engineering
- Strong engineering mindset with a focus on best practices
- Proficiency in Python and SQL
- Solid knowledge of AWS
- Experience with Kubernetes (K8s) and Terraform
Nice if you also have
- Experience with data ingestion pipelines such as Meltano, Airbyte, Stitch/Singer.io, or similar
- Proficiency in developing production-grade ELT/ETL pipelines using tools like Dbt and Airflow
- Familiarity with Data Observability principles
- Experience with streaming data pipelines and technologies like Spark Streaming, Flink, KsqlDB, and Kafka
- Highly desirable to have experience with Spark (Python or Scala)
- Familiarity with technologies like Hudi, DeltaLake, or Iceberg
