Senior Data Engineer (AWS Databricks)

We are seeking a Data Engineer to join our team, focused on data platform integration and pipeline engineering on Databricks. This role will work closely with AI Engineers to enable model development and deployment by building secure, reliable, and scalable data pipelines and integrations.

The role is not focused on data transformation or analytics modelling. Instead, it concentrates on ingestion, orchestration, connectivity, and operational pipelines that support AI and advanced analytics workloads within Databricks environments deployed into client-managed accounts.

Experience Required: 5+ yrs

Key Responsibilities

Databricks-Centric Data Engineering

• Build and maintain data pipelines that ingest data into Databricks on AWS.
• Configure and manage Databricks jobs and workflows to support AI workloads.
• Integrate Databricks with upstream source systems and downstream AI services.
• Ensure data is accessible and performant for AI training and inference use cases.

Pipeline Engineering (Non-Transformational)

• Design and implement pipelines focused on:
• Data ingestion
• Data movement
• Orchestration and scheduling
• Develop pipelines using Python and Databricks-native tooling (e.g. Databricks Jobs, workflows).
• Ensure pipelines are production-ready with monitoring, logging, and alerting.

AWS Environment Integration

• Work within client-owned AWS environments, collaborating with DevOps engineers on infrastructure provisioning.
• Integrate Databricks pipelines with cloud services such as:
• Ensure pipelines align with client security and governance requirements.

Security, Governance & Compliance

• Build and operate pipelines that meet SOC 1 compliance requirements, including:
• Access controls and permissions
• Audit logging and traceability
• Controlled deployment processes
• Support data governance standards within Databricks environments.

Delivery & Operations

• Deploy pipeline code via GitHub-based CI/CD pipelines.
• Support operational monitoring and incident response for data pipelines.
• Document pipeline designs, dependencies, and operational processes.

Core Technical Skills

• Strong experience as a Data Engineer, with a focus on Databricks-based platforms.
• Hands-on experience with Databricks on AWS, including:
• Jobs and workflows
• Cluster configuration (user-level understanding)
• Proficiency in Python for pipeline development.
• Experience using GitHub and CI/CD workflows.

Engineering & Delivery

• Experience building production-grade ingestion and orchestration pipelines.
• Ability to work effectively in client-facing delivery environments.
• Strong documentation and collaboration skills.

Nice to Have

• Experience with Databricks Unity Catalog.
• Exposure to event-driven or streaming data architectures.
• Familiarity with MLOps concepts and AI lifecycle support.