Data Engineer

About the role 

We are seeking a skilled Data Engineer to join our growing Tech Ops division. As the organization continues to expand, this role provides the opportunity to contribute meaningful technical impact across our data delivery ecosystem. In this position, you will collaborate with customers, data vendors, and internal engineering teams to design, implement, and maintain reliable data integration solutions within a modern, cloud‑based environment. 

You will apply advanced knowledge of distributed computing, data architecture, and cloud‑native engineering to build scalable and resilient data ingestion and transformation pipelines. As a technical contributor, you will support the adoption of Abacus’s core data management platform and help ensure high‑quality, compliant data operations across the data lifecycle. 

Your day‑to‑day 

Design and develop data pipelines 

• Build and maintain batch and real‑time data pipelines using PySpark, SparkSQL, Databricks Workflows, and distributed processing frameworks. 

• Create ingestion frameworks integrating with Databricks, Snowflake, AWS services (S3, Lambda, SQS), and vendor data APIs, ensuring data quality, lineage, and schema evolution. 

Support data architecture & modeling 

• Develop data models, including star/snowflake schemas, and apply optimization techniques for analytical workloads in cloud data environments. 

• Participate in design discussions for reliable and fault‑tolerant architectures across multi‑account AWS environments. 

Translate business needs into engineering solutions 

• Convert customer and internal business requirements into clear technical specifications, data flows, and engineering components. 

• Contribute to reusable frameworks and standardized engineering patterns. 

Implement security & compliance workflows 

• Apply data security best practices including RBAC, encryption at rest/in transit, PHI handling, tokenization, auditing, and adherence to HIPAA and SOC 2 compliance requirements. 

• Support automation of secure data operations and governance frameworks. 

Advance engineering best practices 

• Implement CI/CD workflows for data pipelines, including code versioning, automated testing, orchestration, observability, and logging standards. 

• Conduct performance profiling and assist with compute optimization, cost efficiency, and tuning techniques for Databricks and Snowflake environments. 

Documentation & Collaboration 

• Produce clear technical documentation, runbooks, architecture diagrams, and operational standards. 

• Assist senior engineers through collaborative code reviews, knowledge‑sharing sessions, and cross‑team engineering initiatives. 

What you bring to the team 

• Bachelor’s degree in Computer Science, Computer Engineering, or a closely related technical field. 

• 3+ years of hands‑on experience as a Data Engineer working with distributed data processing systems in modern cloud environments. 

• Working knowledge of U.S. healthcare data domains — including claims, eligibility, and provider datasets — and ability to apply this knowledge to ingestion and transformation workflows. 

• Strong communication skills to articulate technical concepts across both technical and non‑technical stakeholders. 

• Proficiency in Python, SQL, and PySpark, including distributed data transformations and performance‑optimized queries. 

• Demonstrated experience building and operating production ETL/ELT pipelines using Databricks, Airflow, or similar orchestration frameworks. 

• Familiarity with cloud‑native data platforms and services such as dbt, Kafka, Delta Lake, and event‑driven/streaming architectures. 

• Experience with structured and semi‑structured data formats (Parquet, JSON, Avro, ORC), including schema evolution techniques. 

• Working knowledge of AWS data ecosystem components—S3, SQS, Lambda, Glue, IAM—used to support scalable data engineering workloads. 

• Experience with Terraform and modern CI/CD pipelines (e.g., GitLab) for automating deployment and versioning. 

• Understanding of SQL optimization strategies including partitioning, pruning, Z‑Ordering, clustering, and caching for large analytical workloads. 

• Experience using cloud data warehouse technologies such as Snowflake, BigQuery, or Redshift, including performance tuning and data modeling practices. 

What we would like to see but not required: 

• Experience in healthcare or payer data environments at scale. 

What you’ll get in return 

• Competitive Leave & Benefits
• Comprehensive health coverage
• Equity for every employee – share in our success
• Growth-focused environment – your development matters here

Working Arrangements

• Standard hours: 9 hours/day, 5 working days
• Location: Onsite
• Shift: 10 AM – 7 PM local time