Consultant/Sr. Consultant, Consumer Analytics Data Engineering

At Lilly, the work is demanding because patients are waiting. We unite caring with discovery to help make life better for people around the world, knowing that every decision, every detail, and every day matters. Headquartered in Indianapolis, Indiana, our over 50,000 employees around the globe take on complex challenges to discover and deliver life-changing medicines, strengthen how health is understood and managed, and support the communities we serve. This is hard, urgent, selfless work—but it’s work worth doing. If you’re driven by purpose and ready to bring your best to work that truly matters for patients, we invite you to join us.  As Eli Lilly strives to achieve its purpose of making life better for patients, we have been building up our in-house ‘Consumer Experience’ function, which designs and executes the next-generation marketing campaigns aimed at informing and educating consumers (or patients) directly. We are seeking a highly skilled Consultant/Sr. Consultant to work with consumer analytics data engineering initiatives at Lilly, Bengaluru. This role is primarily focused on the Databricks platform and AWS ecosystem, designing and maintaining robust data pipelines, lakehouse architectures, and semantic layers that power advanced analytical solutions and AI/agentic workflows for consumer insights. The ideal candidate brings deep hands-on Databricks expertise, good AWS data engineering skills, and a working knowledge of semantic layer design and AI agent development. The role will be a blend of technical expertise along with business/domain integration.

Design, develop, and implement scalable ETL/ELT pipelines for extracting, transforming, and loading consumer data from various sources (e.g., CRM, marketing platforms, DCM, GA4, digital channels), with Databricks/AWS as the primary execution platform. Develop and manage end-to-end solutions on Databricks including Unity Catalog, Delta Live Tables, Databricks Workflows, Databricks SQL, etc.; own platform governance covering schemas, permissions, and data lineage. Design multi-hop lakehouse architectures (Bronze / Silver / Gold) using Delta Lake; optimize Spark compute, cluster configurations, and Auto Loader for performance and cost efficiency. Leverage AWS data services — S3, Glue, Lambda and Redshift — in conjunction with Databricks to build reliable, end-to-end consumer data flows. Architect and optimize data models and schemas to support complex analytical queries and reporting requirements related to consumer behaviour, preferences, and engagement. Publish semantic layers (metrics definitions, certified datasets, business logic) consumed by downstream BI tools and AI agents; build and deploy agentic workflows using Databricks AI Functions or similar frameworks. Ensure data quality, integrity, and governance across all consumer data assets by implementing validation rules, schema evolution controls, and monitoring processes through Unity Catalog. Collaborate with data scientists, business analysts, and marketing teams to understand data needs and translate them into technical data engineering solutions; partner to productionize ML models and feature stores on Databricks. Implement automation for data ingestion, processing, and delivery with a focus on efficiency, reliability, and SLA adherence. Troubleshoot and resolve data-related issues, performing root cause analysis and implementing corrective actions. Stay current with Databricks platform updates, AWS data services, and emerging best practices in data engineering and AI-driven analytics. Job Qualifications: Bachelor's or Master's degree in Computer Science, Engineering, Information Technology, or a related quantitative field. 3-8 years of data engineering experience with hands-on production experience on Databricks. Deep knowledge of Databricks platform architecture — Unity Catalog, Delta Lake, Databricks Workflows, Databricks SQL, and cluster/compute management.