Lead Engineer, Big Data - Databricks - Remote

JOB DESCRIPTION Job Summary Provides lead level engineering support for information technology (IT) big data architecture activities on Microsoft Azure cloud infrastructure. Representing as a Forward Deployed Engineer, responsible for leading the collection, storage, processing, and analysis of large health care payor datasets supporting Medicare/Medicaid operations. Designs enterprise-scale data solutions on Azure Databricks, Azure Data Factory, and Azure Synapse with emphasis on performance, scalability, and high-reliability. Champions AI-first practices, mentors team members, and drives adoption of machine learning and advanced analytics to improve member health outcomes. Partners strategically with business leaders to identify data-driven opportunities and delivers end-to-end solutions.

Essential Job Duties • Operates as a lead Forward Deployed Engineer, driving end-to-end ownership culture and mentoring team members on business problem-solving. • Provides subject matter expertise in Azure data platform architecture (Azure Databricks, Azure Synapse, Azure Data Factory, Azure Data Lake). • Champions AI-first development practices, implementing GitHub Copilot workflows and Azure Machine Learning solutions for predictive health care analytics. • Designs and implements business intelligence (BI) solutions using Power BI, Tableau integrated with Azure data platform. • Analyzes current business practices and identifies opportunities for leveraging Azure data and AI PaaS services to improve Medicare/Medicaid operations. • Builds reusable data quality frameworks and automated data validation using Azure DevOps and quality assurance automation tools. • Leads integration initiatives connecting Azure data platform with low-code platforms (Pega, Salesforce, Sitecore). • Converts business requirements into technical architecture, design and implementation for health care analytics use cases. • Designs analytical data models supporting self-service BI for clinical, financial, and operational stakeholders. • Implements batch and real-time data pipelines for claims processing, quality metrics, and population health analytics. • Partners with business leaders across the organization to identify high-value data opportunities and prioritize initiatives. • Leads technical issue resolution and collaborates with Azure support, infrastructure, and vendor teams. • Supports project planning, scheduling and execution using Agile methodologies. • Ensures HIPAA compliance and health care data governance standards are maintained. • Provides training, mentoring and support to new and existing big data team members on Azure data technologies and Forward Deployed Engineer principles.

Required Qualifications • At least 6 years of data management experience on cloud data platforms, or equivalent combination of relevant education and experience. • Bachelor's degree in IT, computer science, statistics, business, health care, engineering, etc. • Experience with Microsoft Azure data services (Azure Databricks, Azure Data Factory, Azure Synapse, Azure Data Lake Storage, Azure Machine Learning). • Proven experience implementing AI-assisted development and machine learning solutions. • Proficiency in Python, PySpark, Scala, and SQL with quality assurance automation experience. • Experience with real-time streaming architectures (Kafka, Azure Event Hubs, Spark Streaming). • Experience integrating data platforms with low-code systems (Pega, Salesforce). • Health care payor industry experience with Medicare/Medicaid claims, quality metrics (HEDIS, Stars), and regulatory reporting. • Understanding of HIPAA requirements and health care data governance frameworks. • Ability to mentor and develop team members with Forward Deployed Engineer mindset. • Experience with Azure DevOps CI/CD pipelines and Infrastructure as Code (Terraform, ARM templates). •