Site Reliability Engineer, Model Post Training

The mission of Thinking Machines is to build AI that extends human will and judgment.

We're looking for a Site Reliability Engineer (SRE) to drive reliability for Model Post Training end-to-end. Post-training spans several distinct workloads — supervised fine-tuning, reinforcement learning, preference optimization, and distillation — each with its own resource profile, failure signature, and tolerance for interruption or delay. You'll set the reliability strategy across all of them, not just react to what breaks. This is a senior individual-contributor role with real technical authority. You'll work alongside the engineers building Tinker's training and serving infrastructure and the research teams running experiments on top of it, shaping how the platform is architected before problems reach production rather than only after. You'll be trusted to define what reliable means for post-training workloads, push back on launches that put stability or correctness at risk, and lead incident response for issues that span multiple teams and layers of the stack. We're looking for someone who has operated production ML systems before and understands, first-hand, how an infrastructure failure — a bad gradient sync, a stale checkpoint, a silent NaN — can look identical to a research problem until someone with the right instincts tells the difference.

Define and own end-to-end reliability for Model Post Training, from CI/CD flows to production observability and incident response, across SFT, RL, DPO, and distillation workloads. Develop Service Level Objectives for distributed training systems, balancing job completion reliability and scheduling latency against development velocity, and tailoring them to each training method's tolerance for delay, restart, and data loss. Design and implement monitoring and observability across the full training path — data loading, forward/backward passes, optimizer steps, checkpointing, and sampling — with enough signal to distinguish an infrastructure failure from a training or modeling issue. Drive incident response for Model Post Training platform issues, ensuring rapid recovery, thorough incident reviews, and systematic improvements that prevent recurrence. Harden multi-tenant isolation and resource scheduling so that LoRA-based workload co-scheduling maximizes utilization without compromising reliability or data separation. Set technical direction on reliability for new post-training methods as they're adopted onto the platform, partnering with research early so resilience is designed in rather than retrofitted. Build and harden checkpointing and recovery for long-running training jobs, so trainer, optimizer, or sampler failures cost minutes of progress, not hours. Mentor other engineers on production ML reliability practices, and collaborate with security teams to address production vulnerabilities.

Minimum Qualifications 7+ years of experience in distributed systems, cloud infrastructure, or site reliability engineering, including direct on-call ownership of a live production service. Hands-on experience operating production machine learning systems — training, fine-tuning, or inference infrastructure — and reasoning about how infrastructure failures manifest as training or model-quality issues. Proficiency writing software to solve reliability problems, including building tooling and automation, typically in Python and/or a systems language such as Go or Rust. Track record of driving production incident response, postmortems, and systematic reliability improvement, including for complex, cross-team, or ambiguous issues. Strong communication skills and a track record of setting technical direction and coordinating across engineering and research teams.

Deep experience operating production cloud services at scale (e.g., public cloud platforms, internal cloud services). Working knowledge of post-training