Senior Databricks Migration Engineer

Accountabilities:: As a Senior Databricks Migration Engineer, you will serve as a technical authority for the data platform migration, combining hands-on engineering with architecture, optimization, governance, and team enablement. Lead the migration of legacy SQL Server stored procedures and Azure Data Factory pipelines to Databricks Lakehouse and Delta Lake. Translate traditional relational data warehouse architectures into scalable Bronze, Silver, and Gold Lakehouse frameworks. Design reusable ETL/ELT frameworks using PySpark, Delta Live Tables (DLT), and Databricks Workflows. Architect and optimize Gold Layer dimensional models and star schemas to maximize Power BI performance. Optimize Databricks SQL Warehouses for high-concurrency, low-latency Power BI workloads across DirectQuery and Import modes. Implement advanced Delta Lake optimization techniques, including Z-Ordering, data skipping, liquid clustering, and materialized views. Define cluster sizing, auto-scaling, and serverless SQL compute standards to balance performance, reliability, and cost. Build monitoring dashboards to track Databricks Unit (DBU) consumption and identify opportunities for cost optimization. Establish effective partitioning strategies and file-size management practices within Delta Lake. Design and implement data security and governance using Unity Catalog. Enforce row-level and column-level security for Power BI users and internal analysts. Align Lakehouse security with Microsoft Entra ID and enterprise RBAC standards. Lead pair-programming sessions, technical workshops, and code reviews to help teams transition from SQL-centric to Spark-centric development. Produce comprehensive technical documentation, including architecture diagrams, design patterns, and optimization playbooks. Establish knowledge-transfer practices that enable internal teams to operate and enhance the platform independently after migration. Communicate technical concepts effectively to both technical and non-technical audiences. Manage work in an organized and timely manner while maintaining a strong attention to detail. Requirements The role requires strong data engineering fundamentals, hands-on Databricks and cloud experience, and the ability to lead complex technical initiatives while collaborating effectively across teams. Bachelor’s degree or higher from an accredited college or university in Computer Science, Engineering, or a related technical field. 5+ years of experience in data engineering, data system development, or related roles. 5+ years of experience working with cloud platforms such as Azure, AWS, or GCP. At least 1 year of experience leading complex, cross-functional data projects and technical teams. Strong expertise in data engineering principles, data modeling, ETL processes, and data pipeline development. Hands-on experience with Databricks Lakehouse, Apache Spark, Delta Lake, cloud-native databases, cloud storage, and distributed computing platforms. Strong proficiency in SQL and Python/PySpark for data manipulation and pipeline development. Experience with Azure Data Lake storage and processing services. Experience designing, building, and optimizing data pipelines for ingestion, transformation, and loading. Experience with data warehousing, dimensional modeling, enterprise data lakes, incremental data loads, metadata-driven ingestion, and data quality frameworks using PySpark. Ability to identify and resolve complex data-related challenges. Strong understanding of query performance optimization, scalability, and efficient data modeling. Demonstrated ability to communicate effectively in both written and verbal formats with technical and non-technical stakeholders. Strong organizational skills, attention to detail, and ability to complete assignments within established timelines. Must be a U.S. citizen and able to obtain a Position of Public Trust clearance. Must not have traveled outside the United States for a combined total of 6 months or more during the past 5 years and must have resided in the United States for the past 5 years. Benefits Fully remote opportunity available to candidates throughout the continental United States. Preference for candidates located locally, followed by candidates in the U.S. East Coast time zone. Opportunity to contribute to a major data modernization and Lakehouse migration initiative supporting a Federal Government customer. Senior-level technical ownership across architecture, engineering, optimization, security, and governance. Opportunity to influence engineering practices and enable internal teams through mentoring and knowledge transfer. Collaborative environment involving data engineers, analysts, technical teams, and other stakeholders. Position includes the opportunity to work with modern technologies across Databricks, Spark, Delta Lake, Azure, Power BI, and Unity Catalog. Equal opportunity and inclusive workplace environment.