Startup Jobs
Post a Job
MediaRadar

Senior Data Engineer - Databricks

Vadodara , India
Remote

AI overview

Work on building and maintaining scalable data solutions leveraging modern technologies like Azure Databricks and Apache Spark while collaborating with cross-functional teams to support analytics and

MediaRadar, An Industry Leader in Marketing Intelligence now including the data and capabilities of Vivvix, powers the mission-critical marketing and sales decisions that drive competitive advantage. Our next-generation marketing intelligence platform enables clients to achieve peak performance with always-on insights that span the media, creative, and business strategies of 5 million brands across 30+ media channels and 275 billion in media spend. By bringing the advertising past, present, and future into focus, our clients rapidly act on the competitive moves and emerging advertising trends impacting their business.

About the Role

We are looking for an experienced and strategic Senior Data Engineer to join our data team. In this role, you will be responsible for building and maintaining scalable, high-performance data solutions using Azure Databricks, Apache Spark, AKS, Airflow, Postgres, and modern data lakehouse architectures. You’ll play a key role in the full software development lifecycle—from design and implementation to deployment and documentation—while collaborating cross-functionally to support analytics, reporting, and operational data needs. This is an exciting opportunity to work along with a great team of data engineers, demanding technologies and an engaging work environment to help shape our data engineering best practices.

Requirements

What You’ll Do:

  • Involve in Design, development, and maintenance of scalable ETL/ELT pipelines on Azure Databricks using Apache Spark (PySpark/Spark SQL).
  • Design and implement both batch and real-time data ingestion and transformation processes.
  • Build and manage Delta Lake tables, schemas, and data models to support efficient querying and analytics.
  • Consolidate and process large-scale datasets from various structured and semi-structured sources (e.g., JSON, Parquet, Avro).
  • Write optimized SQL queries for large datasets using Spark SQL and PostgreSQL.
  • Develop, schedule, and monitor workflows using Databricks Workflows, Airflow or similar orchestration tools.
  • Design, build, and deploy cloud-native, containerized applications on Azure Kubernetes Service (AKS) and integrate with Azure services.
  • Ensure data quality, governance, and compliance through validation, documentation, and secure practices.
  • Collaborate with data analysts, data architects, and business stakeholders to translate requirements into technical solutions.
  • Contribute to and enforce best practices in data engineering, including version control (Git), CI/CD pipelines, and coding standards.
  • Continuously enhance data systems for improved performance, reliability, and scalability.
  • Mentor junior engineers and help evolve team practices and documentation.
  • Stay up to date on emerging trends, technologies, and best practices in the data engineering space.
  • Work effectively within an agile, cross-functional project team.

What You’ve Done:

  • Proven experience as a Data Engineer, with a strong focus on Azure Databricks and Apache Spark.
  • Proficiency in Python, PySpark, Spark SQL, and working with large-scale datasets in different data formats.
  • Strong experience designing and building ETL/ELT workflows in both batch and streaming environments.
  • Solid understanding of data lakehouse architectures and Delta Lake.
  • Experience in Azure Kubernetes Service (AKS) is desired.
  • Proficient in SQL and experience with PostgreSQL or similar relational databases.
  • Experience with workflow orchestration tools (e.g., Databricks Workflows, Airflow, Azure Data Factory).
  • Familiarity with data governance, quality control, and security best practices.
  • Strong problem-solving skills and attention to detail.
  • Excellent communication and collaboration skills, with a track record of working cross-functionally.
  • Experience mentoring junior engineers and leading by example.
  • Comfortable working in agile development environments and using tools like Git, CI/CD, and issue trackers (e.g., Jira).

Benefits

At MediaRadar, we are committed to creating an inclusive and accessible workplace where everyone can thrive. We believe that diversity of backgrounds, perspectives, and experiences makes us stronger and more innovative. We are proud to be an Equal Opportunity Employer and make employment decisions without regard to race, color, religion, sex (including pregnancy, sexual orientation, or gender identity), national origin, age, disability, genetic information, or any other legally protected status. This is a full-time exempt role with base salary plus benefits. Final compensation will depend on location, skill level, and experience.

MediaRadar

About MediaRadarMediaRadar is an award-winning advertising intelligence solution that is used by media planning, buying, and selling teams.MediaRadar offers comprehensive advertising analysis for over 4 million brands across multiple media platforms including TV, digital, mobile, email, events, social media, and print. Media sellers rely on MediaRadar to prospect, create compelling pitches, and connect with the right buyers. Agencies and marketers use MediaRadar to create the best media mix, uncover new advertising opportunities, and monitor their competitors. MediaRadar also offers custom solutions and data licensing for use in market research, sales intelligence, ad compliance, and equity markets.In November of 2023, MediaRadar acquired Vivvix, a company offering competitive advertising intelligence across both digital and traditional media channels, such as mobile apps, streaming services, and social media. MediaRadar and Vivvix are both AI-powered platforms, with AI used to track and collect advertising insights across media channels, as well as to power sales recommendations and prospecting tools. Following the Transaction, MediaRadar’s expanded resources and capital will allow it to further invest in cutting-edge data and tech capabilities, building next-gen solutions that drive maximum value for its customers and cementing its position as the industry’s analytics leader.Why we do it?Because we can. We’re not kidding! Because our customers are flooded with data, and we’ve got the skills and tools to help. And helping businesses solve problems, answer critical questions with our data, and be delighted with the outcome makes us proud of what we’ve built.The amount of data generated and collected in our world continues to grow exponentially, and as they say, if you’re not on that bus, you’re under it. At MediaRadar, we’ve been collecting, analyzing, and delivering insights distilled from huge amounts of data to publishers and advertisers since 2006. Our clients see us as a solution to their everyday challenges, not just another source of data.Why will you want to work here?If you’re looking for an opportunity to work with other smart, ambitious people, to help build a company that invents market-leading SaaS solutions that our customers rave about, you’ve come to the right place!We strongly value rolling up our sleeves and taking on challenges – and we do it in a fast-paced and fun environment. Get started, get involved, and make your mark: ideas come from everyone – especially newbies!

Website LinkedIn
View all jobs
Ace your job interview

Understand the required skills and qualifications, anticipate the questions you may be asked, and study well-prepared answers using our sample responses.

Senior Data Engineer Q&A's
Report this job
Apply for this job