Senior ML Ops / LLM Ops Engineer

Overview:

This role focuses on building and operating the ML Ops / LLM Ops pipeline that closes it: ingest production signal, redact it, store it, slice it, classify it, surface the failures, mine new eval cases, and alert on regressions. You drive the toolchain decisions, the data-governance posture, and the day-to-day reliability of the pipeline itself. The Head of AI sets vision and priorities and you own the technical execution end-to-end.

What will you do?

• Design and build a source-agnostic ingestion pipeline for production ML / LLM traffic
• Design storage tiering based on automotive and company requirements, policy-driven retention windows, and privacy requirements
• Build slicing dashboards and the query path engineers use to debug production at 11p.m.
• Enable autoraters and lightweight LLM classifiers across production traffic
• Build the rule-based triage layer for obvious failures
• Stand up the eval-mining workflow and wire regression alerts to model and prompt deploys
• Implement PII redaction at the ingestion boundary and safety / abuse classification on inbound content
• Define dashboard architecture, wipeout mechanisms, tool and hosting selection, and operate the pipeline end-to-end

What are we looking for?

Must Have

• Proven experience building and operating data or ML platform systems in production, covering ingest, schema, storage, access control, alerts, and on-call
• Hands-on experience building and running ML / LLM evaluation systems in production (offline regression sets, online autoraters, LLM-as-judge pipelines, golden datasets)
• Hands-on experience with LLM tracing and observability tooling
• Experience shipping PII redaction or comparable data-handling controls in a regulated or multi-tenant environment, with a pragmatic approach to data governance
• Strong understanding of how ML and LLM-based systems fail in production: hallucination, retrieval failures, agent loops that don’t terminate, ASR / TTS degradation, and prompt or model regressions across deploys
• Production Python proficiency; hands-on engineer, not advisory. Comfortable leveraging AI in everything you build

Nice to Have

• Preferable multi-tenant or white-label SaaS experience with per-tenant data isolation
• Azure experience and ability to make self-host vs managed SaaS calls on tradeoffs
• Experience with autorater methodology and contamination defenses
• Knowledge of vector databases, embedding-based clustering or unsupervised failure-mode discovery
• Experience with data-versioning tooling (LakeFS, DVC, Delta Lake)
• GDPR / right-to-erasure work
• Embedded, automotive, or another constrained environment context
• Working knowledge of a language beyond English sufficient to validate non-English failure modes
• Prior experience using Cloud (Microsoft Azure and AWS);
• Prior experience with Claude Code;
• Prior experience with GitHub;
• Languages: Python primary, SQL, and some TypeScript for dashboards;
• LLM APIs: Claude (Anthropic), OpenAI, open-source models as needed
• Android/AAOS ecosystem as clients