Operations Engineer, Fleet Reliability

fal is the generative media ecosystem powering the next generation of AI products. We build the infrastructure, tools, and model access that teams need to move from idea to production, and do it at scale without compromise. For developers and enterprises, fal is the foundation that makes generative media not just possible, but practical: a unified platform where high-performance inference, orchestration, and observability come together to unlock new categories of AI-native products.

As generative media reshapes industries across a market projected to grow by hundreds of billions over the next decade, fal is becoming the ecosystem that ambitious teams build on.

About the role

As we bring up owned clusters alongside our cloud capacity, we're hiring Operations Engineers to keep the fleet alive. This is a hands-on role. You're first in line when nodes go bad, GPUs throw ECC errors, IB links flap, or a rack stops responding. You'll provision new nodes, validate them, ship them to production, and troubleshoot whatever entropy throws at them. You'll be on-call. You'll be in the weeds.

You're a fit if you've:

• Administered Linux Systems in the critical path before
• Troubleshooted GPU node issues: NVLink, NCCL, IB, driver and firmware bugs
• Has experience in observability systems like Grafana and Prometheus
• Scripted your way out of repetitive work (bash, python, go, whatever)

Who you are:

• Curious. You don't accept "it's flaky" as a root cause
• Comfortable with ambiguity. The runbook doesn't exist yet for half of what you'll do
• On-call doesn't scare you
• You'd rather automate a problem than fix it twice

Responsibilities:

• Provision, validate, and triage GPU nodes across B300, H200, and H100 clusters
• Troubleshoot hardware and software issues across compute, network, and storage
• Monitor fleet health, take remediation action, push fixes upstream when needed
• Write the runbooks. Improve the ones that exist. Delete the ones that don't work