Site Reliability Engineer, RL Infra

The mission of Thinking Machines is to build AI that extends human will and judgment.

We're looking for a Site Reliability Engineer (SRE) to drive reliability for RL Infra end-to-end. RL training is unlike a typical batch training job: it interleaves rollout generation, environment or tool interactions, reward scoring, and policy weight updates in a continuous loop, often across long-tailed trajectories with unpredictable latency. Keeping this loop healthy — and keeping it healthy for many concurrent tenants sharing the same underlying clusters — is the core of this role. You'll work alongside the engineers building the RL training and rollout systems, and with the research teams running experiments on top of them, to make every layer of the stack more robust and resilient: from the rollout and inference-serving layer, through weight synchronization between trainers and samplers, to the schedulers deciding how GPUs are shared across simultaneous jobs. This is a role with real ownership. You'll be trusted to set reliability priorities, push back on changes that put stability or correctness at risk, and drive incidents affecting RL workloads to resolution without waiting for permission.

Define and own end-to-end reliability for RL Infra, from CI/CD flows through production observability and incident response. Develop Service Level Objectives for distributed RL training systems, balancing job completion reliability and rollout/scheduling latency against development velocity. Design and implement monitoring and observability across the full RL loop — rollout generation, environment and tool execution, reward computation, and weight synchronization between trainers and samplers — so failures and slowdowns are caught close to their source. Drive incident response for RL Infra platform issues, ensuring rapid recovery, thorough incident reviews, and systematic improvements that prevent recurrence. Harden multi-tenant isolation and resource scheduling so that LoRA-based workload co-scheduling maximizes utilization without compromising reliability or data separation. Reduce the impact of long-tailed rollouts and stale or off-policy data on training stability, working with research and infra teams to bound staleness and recover cleanly from stuck or slow trajectories. Build and harden checkpointing and recovery for long-running RL jobs, so trainer, sampler, or environment failures cost minutes of progress, not hours. Collaborate with security teams to address production vulnerabilities across the RL Infra stack.

Minimum Qualifications Bachelor's degree or equivalent experience in computer science, engineering, or a similar field. Experience in distributed systems, cloud infrastructure, or site reliability engineering. Proficiency writing software to solve reliability problems, including building tooling and automation. Experience with production incident response, postmortems, and systematic reliability improvement. Strong communication skills and a track record of coordination across engineering and research teams.

Deep experience operating production cloud services at scale (e.g., public cloud platforms, internal cloud services). Background in distributed training frameworks and how infrastructure failures — stuck rollouts, stale weights, environment flakiness — surface in training behavior. Direct experience with RL-specific infrastructure: rollout/inference serving, reward pipelines, or asynchronous training systems that overlap generation with policy updates. Track record building checkpoint and recovery systems for long-running distributed jobs. Expertise in Kubernetes at scale: deploying, operating, debugging, and tuning clusters handling heterogeneous GPU workloads. Logistics Location: This role is based in San Francisco, California.

Depending on background, skills and experience, the expected annual salary range for this