Data Engineer

Vadodara , India
Hybrid

AI overview

Architect and build scalable ELT/ETL pipelines using Databricks and Spark, ensuring data integrity and optimizing performance in a collaborative Agile environment.

MediaRadar, now including the data and capabilities of Vivvix, powers the mission-critical marketing and sales decisions that drive competitive advantage. Our competitive advertising intelligence platform enables clients to achieve peak performance with always-on data and insights that span the media, creative, and business strategies of five million brands across 30+ media channels. By bringing the advertising past, present, and future into focus, our clients rapidly act on the competitive moves and emerging advertising trends impacting their business. 

We are seeking an experienced Data Engineer to join the team and take ownership of our data infrastructure. With millions of new data points ingested daily, your mission will be to architect, build, and scale robust data pipelines that ensure flawless data quality. You'll work with a passionate team on a modern cloud data stack (Azure, Databricks), solving complex challenges to deliver timely and reliable data that drives our business and delights our customers.

What You'll Do

Architect & Build: Design, implement, and optimize scalable and reliable ELT/ETL pipelines using Databricks, Spark, Python and Store Procedures. You will take the lead in orchestrating complex workflows using Apache Airflow, ensuring seamless dependency management across our cloud environment.

Ensure Data Integrity: Develop and implement comprehensive testing frameworks, data validation rules, and QA plans to guarantee the accuracy and integrity of our data assets.

Optimize & Troubleshoot: Proactively monitor system performance, tune complex SQL queries, and troubleshoot production issues. You will perform root-cause analysis and implement lasting solutions to improve system health and reliability.

Collaborate & Innovate: Actively participate in an Agile environment (Scrum, sprints, backlog grooming) and collaborate with cross-functional teams. You'll help evaluate and introduce new technologies and best practices to continuously improve our data platform.

Requirements

What You'll Bring (Qualifications)

  • A bachelor’s or master’s degree in computer science, Engineering, or a related field (or equivalent practical experience).
  •  5+ years of hands-on experience in data engineering, with a strong focus on Databricks, Apache Spark and 1 RDBMS solution for processing large-scale datasets.
  •  Proven experience with Apache Airflow for workflow orchestration, including designing DAGs, managing task dependencies, and integrating with Azure-based services.
  • Expert-level proficiency in Python (including PySpark) and a strong understanding of data structures, algorithms, and OOP principles.
  • Deep expertise in SQL and Spark SQL, with proven experience writing and optimizing complex analytical queries.
  • Hands-on experience with the Databricks ecosystem, including Delta Lake, Unity Catalog, and Data frame APIs.

Bonus Points (Nice to Haves)

  • Experience building and maintaining CI/CD pipelines using tools like Azure DevOps.
  • Experience with Managed Workflows for Apache Airflow (MWAA) or running Airflow on Kubernetes.
  •  Previous experience migrating a legacy data system to a modern, unified data platform.
  • Experience working in a fast-paced, product-driven environment.

Benefits

At MediaRadar, we are committed to creating an inclusive and accessible workplace where everyone can thrive. We believe that diversity of backgrounds, perspectives, and experiences makes us stronger and more innovative. We are proud to be an Equal Opportunity Employer and make employment decisions without regard to race, color, religion, sex (including pregnancy, sexual orientation, or gender identity), national origin, age, disability, genetic information, or any other legally protected status. This is a full-time exempt role with base salary plus benefits. Final compensation will depend on location, skill level, and experience.

About MediaRadarMediaRadar is an award-winning advertising intelligence solution that is used by media planning, buying, and selling teams.MediaRadar offers comprehensive advertising analysis for over 4 million brands across multiple media platforms including TV, digital, mobile, email, events, social media, and print. Media sellers rely on MediaRadar to prospect, create compelling pitches, and connect with the right buyers. Agencies and marketers use MediaRadar to create the best media mix, uncover new advertising opportunities, and monitor their competitors. MediaRadar also offers custom solutions and data licensing for use in market research, sales intelligence, ad compliance, and equity markets.In November of 2023, MediaRadar acquired Vivvix, a company offering competitive advertising intelligence across both digital and traditional media channels, such as mobile apps, streaming services, and social media. MediaRadar and Vivvix are both AI-powered platforms, with AI used to track and collect advertising insights across media channels, as well as to power sales recommendations and prospecting tools. Following the Transaction, MediaRadar’s expanded resources and capital will allow it to further invest in cutting-edge data and tech capabilities, building next-gen solutions that drive maximum value for its customers and cementing its position as the industry’s analytics leader.Why we do it?Because we can. We’re not kidding! Because our customers are flooded with data, and we’ve got the skills and tools to help. And helping businesses solve problems, answer critical questions with our data, and be delighted with the outcome makes us proud of what we’ve built.The amount of data generated and collected in our world continues to grow exponentially, and as they say, if you’re not on that bus, you’re under it. At MediaRadar, we’ve been collecting, analyzing, and delivering insights distilled from huge amounts of data to publishers and advertisers since 2006. Our clients see us as a solution to their everyday challenges, not just another source of data.Why will you want to work here?If you’re looking for an opportunity to work with other smart, ambitious people, to help build a company that invents market-leading SaaS solutions that our customers rave about, you’ve come to the right place!We strongly value rolling up our sleeves and taking on challenges – and we do it in a fast-paced and fun environment. Get started, get involved, and make your mark: ideas come from everyone – especially newbies!

View all jobs
Ace your job interview

Understand the required skills and qualifications, anticipate the questions you may be asked, and study well-prepared answers using our sample responses.

Data Engineer Q&A's
Report this job
Apply for this job