Software Engineer L5 - AI Observability & Agent Evaluation

At Netflix, our mission is to entertain the world. Together, we are writing the next episode - pushing the boundaries of storytelling, global fandom and making the unimaginable a reality. We are a dream team obsessed with the uncomfortable excitement of discovering what happens when you merge creativity, intuition and cutting-edge technology. Come be a part of what’s next.

AI and ML power innovation in all areas of the business, including helping members choose the right title for them through personalization, better understanding our audience and our content slate, creating high-quality subtitles, dubbings, images, trailers, and other assets, optimizing our payment processing, and much more. AI Platform (AIP) organization builds highly scalable, differentiated AI infrastructure to maximize the business impact of all AI/ML practitioners at Netflix, which is key to accelerating this innovation.

The AI Observability team makes AI, ML, and Agentic systems transparent, reliable, and production-ready at scale. We build end-to-end observability for ML and GenAI workloads, capturing model inputs, features, predictions, outcomes, and behavior across online and batch systems. Our platform enables teams to monitor model performance, data quality, drift, latency, and failures, turning the ML system from a black box into an explainable, debuggable system. We provide developer-friendly libraries, dashboards, and alerts so teams can debug issues, respond to incidents, and ship AI-powered products with confidence.

We're looking for a hands-on senior engineer to build the frameworks behind Netflix's AI Observability platform, model performance, evaluation, and vendor integration surfaces. You will design reusable infrastructure that enables ML/AI practitioners across domains to monitor model quality in production, evaluate LLM and agentic systems, and adopt vendor tooling through a consistent, self-serve platform. AIP owns the generic, reusable infrastructure; domain teams own their domain-specific evals and remediation. You will partner closely with engineering, product, machine learning, and data teams to turn their needs into reusable platform capabilities. To succeed, you will bring a strong background in AI or ML infrastructure and a passion for building scalable, robust systems.

* Build the observability framework and platform capabilities that give ML and GenAI systems metrics, logs, and distributed traces across online inference, batch scoring, feature pipelines, and agent orchestration, so teams can instrument their systems consistently.

* Build the primitives that let teams monitor model performance (accuracy, calibration, error rates), data quality, drift, and degradation on their own systems, rather than monitoring individual models yourself.

* Build and extend evaluation frameworks for LLM and agentic systems that support response quality, grounding and hallucination, task success, tool-use and trajectory correctness, and LLM-as-a-judge and human-in-the-loop scoring, giving teams reusable building blocks to define and run their own evals.

* Lead build-vs-buy evaluations for observability and eval tooling, and own the SDKs, connectors, and APIs that integrate vendor platforms into a consistent, well-supported interface, so ML and product teams can onboard models and agents with minimal friction.

* Build reusable libraries, SDKs, and templates that make observability and evaluation the default for new systems ("observability-by-default"), lowering the barrier for teams to instrument and evaluate their work.

* Provide the dashboarding, alerting, and SLO/SLI building blocks (plus sensible out-of-the-box templates) that teams use to track model performance, latency, cost, and reliability.

* Experience in software, AI/ML, or platform engineering, with hands-on time in production observability, monitoring, or ML/LLM evaluation

* Proven