We are seeking a highly motivated and skilled Quality Engineer to join our Data Engineering team in Hyderabad, India. This individual will play a critical role in ensuring the accuracy, reliability, and integrity of our central data lake and reporting products. The ideal candidate will be passionate about data quality, possess a strong understanding of data engineering principles, and be adept at developing and implementing robust quality controls and testing strategies for data pipelines and reporting solutions.
Key Responsibilities
-
Design and Execute Quality Strategy: Develop and implement comprehensive test strategies and quality control processes specifically for data ingestion, transformation, storage, and consumption within the data lake and reporting ecosystem.
-
Data Quality Testing: Create, execute, and maintain automated and manual data quality tests (including unit, integration, and end-to-end tests) to validate data accuracy, completeness, consistency, and conformity with business rules.
-
Pipeline Validation: Validate ETL/ELT data pipelines to ensure data is processed correctly and efficiently, focusing on error handling, performance, and scalability.
-
Reporting and Dashboard Verification: Thoroughly test and validate the data displayed in business intelligence (BI) reports and dashboards against source data to guarantee accuracy and reliability for business decision-making.
-
Quality Gates & Monitoring: Implement quality gates within the CI/CD pipeline and establish data quality monitoring dashboards to proactively identify and alert on data anomalies and pipeline failures.
-
Collaboration: Work closely with Data Engineers, Data Scientists, and Product Managers to understand data requirements, define quality metrics, and integrate quality assurance throughout the data development lifecycle.
-
Documentation: Document test plans, test cases, and quality control processes, as well as track and report on data quality metrics and issues.
Required Qualifications
- 5+ years of experience in Quality Assurance or Quality Engineering, with a focus on data-centric testing or data warehousing environments.
- Strong proficiency in SQL for data querying and validation.
- Experience in developing test automation frameworks and scripts for data pipelines using tools/languages like Python, PHP, or Javascript.
- Solid understanding of cloud-based data warehousing and data lake concepts (e.g., AWS S3/Glue, Azure Data Lake/Synapse, Google Cloud Storage/BigQuery).
- Familiarity with ETL/ELT tools and data pipeline orchestration (e.g., Apache Airflow, dbt, Talend).
- Experience with BI tools (e.g., Tableau, Looker, Power BI) for report validation and testing.
Preferred Qualifications
- Experience with big data technologies.
- Knowledge of data governance, metadata management, and master data management principles.
- Familiarity with performance testing and optimization techniques for large-scale data systems.
- Experience working in an Agile/Scrum development environment.
- Technical leadership for steering testing efforts.
