AT&T Databricks - Lead

We are looking for a Senior Lead – Databricks & Data Modernization to lead the design and implementation of a data modernization program involving both new data feature development in Databricks and the migration of existing capabilities from Palantir Foundry to Databricks . The role requires a strong combination of Databricks engineering, data architecture, technical leadership and program execution skills . The candidate will be responsible for translating business and technical requirements into scalable Databricks solutions, guiding the engineering team, driving the Palantir-to-Databricks migration, and ensuring high-quality delivery. The Senior Lead will work closely with customer stakeholders, enterprise/data architects, product owners, engineering teams and AI/ML teams and will be expected to take ownership of technical decisions and end-to-end delivery.

1. Technical Leadership & Solution Design Lead the design and development of scalable Databricks-based data solutions . Translate business requirements into technical architecture, data models and implementation plans. Define appropriate patterns for data ingestion, transformation, processing, storage and serving. Establish engineering standards, coding standards and reusable frameworks for the Databricks development team. Make technology and design decisions balancing performance, scalability, maintainability, security and cost . Review technical designs and code produced by the engineering team. Identify opportunities to modernize and simplify existing solutions. 2. Palantir Foundry → Databricks Migration Lead the technical assessment and migration of existing Palantir Foundry capabilities to Databricks . Analyze existing Foundry pipelines, datasets, transformations, business rules and dependencies. Develop migration strategies and target-state architectures for different Foundry capabilities. Define appropriate Databricks-native replacements rather than simply performing a one-to-one migration. Identify migration complexity, dependencies, risks and technical gaps. Develop migration plans covering: Data ingestion Data transformation Data quality Data comparison/reconciliation Data models APIs/integrations Downstream analytics and applications Lead source-to-target mapping and functional/data validation. Establish approaches for achieving data and functional parity between Palantir and Databricks. Identify opportunities to improve performance, scalability and maintainability during migration. 3. Databricks Engineering Provide hands-on technical leadership using: Databricks Apache Spark PySpark Spark SQL Python Delta Lake Unity Catalog Lakeflow / Spark Declarative Pipelines Databricks Workflows Design and implement batch and near-real-time data pipelines. Implement incremental processing, CDC and SCD patterns. Design and optimize Medallion Architecture implementations. Implement data quality, validation, reconciliation and error-handling frameworks. Optimize Spark workloads and Delta tables for performance and cost. Ensure production-grade engineering practices across the team. 4. Data Architecture & Governance Work with enterprise architects to define the target Databricks data architecture . Define appropriate Bronze, Silver and Gold data layers. Establish data modelling and data integration standards. Ensure appropriate use of Unity Catalog for governance, access control and lineage. Define standards for data quality, metadata, lineage and observability. Ensure solutions comply with enterprise security and governance requirements. 5.

Leadership & Mentoring Lead and mentor a team of Databricks/data engineers. Break down requirements into technical tasks and assign work across the team. Provide technical guidance and unblock engineers on complex issues. Conduct design and code reviews. Establish engineering best practices and reusable components. Support capability development and knowledge sharing within the team. Act as the primary