Senior Software Engineer - Compute

Aurora hires talented people with diverse backgrounds who are ready to help build a transportation ecosystem that will make our roads safer, get crucial goods where they need to go, and make mobility more efficient and accessible for all. 

The journey to developing our cutting-edge self-driving truck Driver software requires a massive and continuous computational effort. Every day, Aurora's engineers initiate and manage millions of compute tasks, systematically processing and analyzing petabytes of critical data. This colossal computational workload spans various essential domains, including raw and derived data processing pipelines, large-scale simulations to test and validate the software in countless scenarios, and the intensive machine learning training jobs that are the core of our autonomous system's intelligence.

At the heart of orchestrating this monumental compute infrastructure is the Compute team. This team is dedicated to building and maintaining the foundational technology that solves the fundamental challenges of resource scheduling, task isolation, and distributed state consistency across our massive batch compute fleet. At our scale, traditional off-the-shelf orchestrators break. The Compute team builds the custom engine - BatchAPI - that manages the lifecycle of millions of tasks - built on top of K8s primitives but implements our own custom scheduler.

We deal with the 'unsolved' problems of distributed computing: maximizing hardware utilization while ensuring that a failure in one node doesn't cascade across the entire cluster. This engine is engineered to handle massive scale, ensuring reliability, efficiency, and rapid turnaround for our engineers.

Furthermore, the Compute team empowers engineers across the company to effectively harness this compute power. They develop and maintain the Batch Workflows Python SDK. This framework provides an intuitive, high-level interface that allows engineers to programmatically define, construct, deploy, monitor, and manage their complex computational workloads. The SDK abstracts away the complexities of the underlying infrastructure, enabling engineers to focus purely on the logic and goals of their data processing, simulation, or training tasks, thus accelerating the entire development cycle for the Aurora Driver.

In this role you will

• Design, implement, and maintain core components of the high-performance, large-scale distributed batch compute engine (BatchAPI). Architect and optimize the scheduler, resource allocator, and execution engine of BatchAPI to handle bursty, heterogeneous workloads with minimal overhead.
• Design low-latency APIs and resilient communication protocols that bridge our Python SDK with the Golang-based core engine.
• Develop high-level workflow abstractions, enabling engineers across the company to programmatically define, deploy, and manage complex data processing, simulation, and ML training pipelines. 
• Solve complex problems in distributed locking, throttling, and fair-share scheduling to ensure multi-tenant stability.
• Drive continuous improvements in the performance, scalability, and resilience of the entire compute infrastructure, implementing robust monitoring and alerting systems to maintain operational excellence for critical workflows.
• Collaborate closely with infrastructure and product engineering teams (e.g., Autonomy, Data, Simulation, Machine Learning) to gather requirements, provide expert consultation, and integrate compute workflows with key company systems.

Required qualifications

• 5+ years of professional software engineering experience.
• Deep expertise in Golang (for core systems) and Python (for SDK/API layering).
• Strong understanding of distributed systems fundamentals (e.g., CAP theorem, consensus algorithms, or gossip protocols).
• Experience with performance profiling and tuning (e.g., memory management, I/O bottlenecks, or network latency optimization).
• Specialized knowledge of container orchestration systems like Kubernetes.
• Proven track record of driving continuous performance, scalability, and resilience improvements in production environments managing critical data.
• Familiarity with cloud provider compute and data services (e.g., AWS EKS, S3, RDS).

Desirable qualifications

• Experience working with computational workloads specific to the autonomous vehicle, robotics, or large-scale machine learning domains (e.g., data processing for perception, simulation, or model training).
• Demonstrated ability in creating and refining user-facing tools, including adeptness at incorporating user feedback, managing expectations, and effectively prioritizing development based on user needs.
• Web UI development experience (Typescript, React)

 #LI- #Mid-Senior 

The base salary range for this position is  $162,000 - $260,000 per year. Aurora’s pay ranges are determined by role, level, and location. Within the range, the successful candidate’s starting base pay will be determined based on factors including job-related skills, experience, qualifications, relevant education or training, and market conditions. These ranges may be modified in the future. The successful candidate will also be eligible for an annual bonus, equity compensation, and benefits.