Databricks Data Engineer and Scientist

Accountabilities: Design, build, deploy, and maintain end-to-end data pipelines for an enterprise Lakehouse using Databricks and AWS. Develop and manage reliable ETL processes with comprehensive logging, error handling, troubleshooting, and issue resolution. Design and maintain data models within a Bronze, Silver, and Gold medallion architecture, selecting appropriate dimensional or relational modeling approaches. Analyze and decompose complex legacy SQL, rebuild pipelines and data models, and validate results for accuracy and performance. Optimize data pipeline performance, scalability, and query execution using partitioning, bucketing, caching, and other relevant techniques. Use AI-assisted coding tools to accelerate pipeline analysis, data engineering, debugging, and data modeling tasks. Support data security and compliance requirements while collaborating with cross-functional teams to deliver high-quality, analytics-ready data. Requirements Bachelor's degree in Computer Science, Computer Engineering, Engineering, Mathematics, Physics, or Information Systems and Technology. At least 8 years of experience designing, developing, deploying, and managing ETL processes and data pipelines. At least 3 years of experience building, debugging, and optimizing data pipelines using Databricks notebooks, clusters, and job orchestration tools, including Delta Lake and Unity Catalog. At least 3 years of experience developing Python and PySpark code for ETL scripts and custom data transformations. Practical experience using AI-assisted coding tools, such as Claude Code or Cursor, to support data engineering and data modeling activities. Experience designing data models using dimensional and relational approaches within a Bronze, Silver, and Gold medallion architecture. Hands-on experience with AWS services, including AWS Glue, Amazon S3, and Amazon RDS. Knowledge of ETL and query performance optimization techniques, including partitioning, bucketing, and caching. Demonstrated ability to analyze complex legacy SQL, reconstruct data pipelines and models, and compare outputs to validate accuracy and performance. Ability to obtain and maintain a Public Trust or other required Suitability/Fitness determination, depending on client requirements. U.S. citizenship is required to support the federal contract. Benefits Work arrangement: Remote position based in the United States. Employment type: Full-time. Technical environment: Opportunity to work with Databricks, AWS, Lakehouse architecture, Python, PySpark, and AI-assisted development tools. Compensation: Salary, bonuses, and other financial benefits were not specified in the provided job description and should be confirmed during the hiring process. Additional benefits: Healthcare, paid time off, retirement plans, and other employee perks were not detailed in the provided description and should be confirmed with the hiring team.