Test Automation Lead – AI Testing

At Lilly, the work is demanding because patients are waiting. We unite caring with discovery to help make life better for people around the world, knowing that every decision, every detail, and every day matters. Headquartered in Indianapolis, Indiana, our over 50,000 employees around the globe take on complex challenges to discover and deliver life-changing medicines, strengthen how health is understood and managed, and support the communities we serve. This is hard, urgent, selfless work—but it’s work worth doing. If you’re driven by purpose and ready to bring your best to work that truly matters for patients, we invite you to join us.  Are you passionate about leveraging technology to automate and transform business processes? Do you have a deep understanding of AI and test automation and a desire to drive innovation in enterprise systems? If so, the Tech Enablement and Automation team at Tech@Lilly is looking for a Lead Consultant in Test Automation to help scale our automation capabilities and quality practices across the organization. As a Lead consultant in Test Automation, you will be responsible for driving the quality and reliability of automation solutions across the enterprise. In this role, you will work closely with Engineers and cross-functional teams throughout the full application lifecycle—ensuring that systems are thoroughly tested, resilient, and aligned with business needs from development through to production.  Are you innovative and curious about new technology that can make business processes easy and more efficient? Come join our team!  What You’ll Be Doing: As a Lead Consultant you will play a pivotal role in shaping and advancing the company’s enterprise-wide quality assurance and engineering strategy for AI powered systems and solutions. You will leverage both your technical expertise and leadership capabilities to oversee the delivery of scalable, high-quality AI and automation solutions. Your responsibilities will include driving resolution of complex testing challenges, leading the implementation of critical enhancements, and spearheading modernization initiatives to ensure our AI and automation frameworks and practices are future-ready and aligned with business objectives.

Deep expertise in test automation frameworks and strategies, with the ability to extend them to validate AI and ML models, data pipelines, and intelligent applications. Hands-on expertise with AI evaluation methodologies , model benchmarking, and quality metrics A strong understanding of AI testing methodologies such as model validation, bias detection, explainability testing, and performance benchmarking . Ability to design end-to-end automated testing solutions for AI-enabled systems involving APIs, UI, data, and model outputs. Proven experience collaborating with data scientists, ML engineers, and software developers to ensure test coverage and model reliability. How You Will Succeed: Lead with influence by promoting a culture of engineering excellence , responsible AI , and continuous improvement. Develop and maintain automation frameworks tailored for testing AI/ML workflows, including model training, inference, and integration . Create and execute automated regression, performance, and data validation tests to ensure the stability and quality of AI-driven systems. Implement AI-specific test approaches such as synthetic data generation, result explainability validation, and continuous model monitoring. Drive innovation by identifying opportunities to embed automation and intelligence into the testing lifecycle using GenAI tools and frameworks. Partner cross-functionally to define and track AI quality metrics (accuracy, precision, recall, fairness) and integrate them into CI/CD pipelines. What You Should Bring: Deep understanding of LLMs, Generative AI, RAG, and Agentic AI architectures Hands-on experience in AI evaluation frameworks ( RAGAS, DeepEval, LLM-as-a-Judge, LangFuse )