About Brillio:
Brillio is one of the fastest growing digital technology service providers and a partner of choice for many Fortune 1000 companies seeking to turn disruption into a competitive advantage through innovative digital adoption. Brillio, renowned for its world-class professionals, referred to as "Brillians", distinguishes itself through their capacity to seamlessly integrate cutting-edge digital and design thinking skills with an unwavering dedication to client satisfaction.
Brillio takes pride in its status as an employer of choice, consistently attracting the most exceptional and talented individuals due to its unwavering emphasis on contemporary, groundbreaking technologies, and exclusive digital projects. Brillio's relentless commitment to providing an exceptional experience to its Brillians and nurturing their full potential consistently garners them the Great Place to Work® certification year after year.
Senior Data Engineer
Primary Skills
Spark - Pyspark, SQL, SQL (Basic + Advanced), Python, Hive, Data Modelling Fundamentals
Specialization
Data Analysis: Data Engineer
Job requirements
3+ years of hands-on experience in production support for data engineering or analytics applications.
Proficiency in PySpark for managing and troubleshooting data processing jobs.
Solid experience with AWS cloud services, including EKS (Elastic Kubernetes Service) and EMR (Elastic MapReduce), S3, Glue, Step Function.
Hands-on experience with Control-M for workflow orchestration and monitoring.
Working knowledge of MongoDB and Snowflake for production data management.
Experience with log monitoring and analysis using Elasticsearch Service (ESS).
Good understanding of ETL processes, incident management, and production monitoring best practices.
Strong problem-solving, troubleshooting, and analytical skills.
Excellent communication and collaboration abilities.
Responsibilities
Provide production support for data pipelines and workflows, ensuring high availability and timely resolution of incidents.
Monitor, troubleshoot, and optimize PySpark-based data processing jobs.
Support and maintain data processing applications deployed on AWS services, including EKS and EMR.
Manage and orchestrate workflows using Control-M, including scheduling, monitoring, and error handling.
Perform root cause analysis and implement permanent fixes for recurring production issues.
Oversee data storage and management in MongoDB and Snowflake, ensuring data quality and consistency.
Monitor and analyze application and system logs using Elasticsearch Service (ESS) to proactively identify and resolve issues.
Collaborate with data engineering, analytics, and operations teams to deliver reliable data solutions.
Maintain and update technical documentation related to production support processes and incident management.
Participate in on-call rotations and respond to critical production alerts as needed.
Know what it’s like to work and grow at Brillio: Click here
