Shape the utilities market of the future with us!
We are looking for an experienced
Senior Data Engineer to design, build, and operate a scalable
integration hub and data pipeline platform using modern open-source technologies. This role is hands-on and suited for an individual contributor who has delivered multiple
end-to-end data engineering projects, from requirements through production deployment and operations.
You will play a critical role in enabling reliable data movement and integration across a diverse ecosystem of third-party systems, APIs, files, databases, and software platforms.
What is the role about?
Key Responsibilities
Data Pipeline Design & Development
- Design, build, and maintain robust ETL/ELT data pipelines for batch and streaming workloads.
- Implement data ingestion and transformation workflows using Apache Airflow, Apache NiFi, Apache Spark, and Kafka.
- Integrate data from multiple sources including REST APIs, files, relational databases, message queues, and external SaaS platforms.
- Optimize pipelines for performance, scalability, reliability, and cost efficiency.
Integration Hub & Platform Engineering
- Develop and operate a centralized data integration hub that supports multiple upstream and downstream systems.
- Build reusable, modular integration components and frameworks.
- Ensure high availability, fault tolerance, and observability of data workflows.
Data Infrastructure & Storage
- Design and manage data warehouses, data lakes, and operational data stores using PostgreSQL and related technologies.
- Implement appropriate data modeling strategies for analytical and operational use cases.
- Manage schema evolution, metadata, and versioning.
Data Quality, Security & Governance
- Implement data validation, monitoring, and reconciliation mechanisms to ensure data accuracy and completeness.
- Enforce data security best practices, access controls, and compliance with internal governance policies.
- Establish logging, alerting, and auditability across pipelines.
Automation & Operations
- Automate data workflows, deployments, and operational processes to support scale and reliability.
- Monitor pipelines proactively and troubleshoot production issues.
- Improve CI/CD practices for data engineering workflows.
Collaboration & Stakeholder Engagement
- Work closely with data scientists, analysts, backend engineers, and business stakeholders to understand data requirements.
- Translate business needs into technical data solutions.
- Provide technical guidance and best practices across teams.
Required skills and qualifications
-
5+ years of hands-on experience as a Data Engineer or in a similar role.
- Proven experience as an individual contributor on at least three end-to-end data engineering projects, from design to production.
- Strong hands-on experience with:
- Apache Airflow / Dagster
- Apache NiFi
- Apache Spark
- Apache Kafka
- PostgreSQL
- Extensive experience integrating data from APIs, files, databases, and third-party systems.
- Strong SQL skills and experience with data modeling.
- Solid programming experience in Python and/or Java/Scala.
- Experience with Linux environments and version control systems (Git).
- Strong problem-solving, debugging, and performance-tuning skills.
Preferred skills and qualifications
- Experience with cloud platforms (AWS, Azure, or GCP).
- Experience with containerization and orchestration (Docker, Kubernetes).
- Knowledge of data lake technologies and formats (Parquet, ORC, Iceberg, Delta Lake).
- Familiarity with monitoring and observability tools for data pipelines.
- Experience working in fast-paced or startup environments.
