Solution Architect, Delivery (Databricks)

Accountabilities:: Enterprise Architecture and Solution Design: Lead the architecture and design of scalable, production-ready Lakehouse solutions using the Databricks Data Intelligence Platform, including Medallion Architecture (Bronze, Silver, and Gold layers), Delta Lake, and Databricks SQL. Data Governance and Security: Design and implement enterprise-wide governance and security frameworks using Unity Catalog, row- and column-level access controls, data masking, attribute-based access control (ABAC), and Delta Sharing. Platform Modernization and Migration: Define and lead technical strategies for migrating legacy enterprise data warehouses and Hadoop environments, including Teradata, Netezza, Snowflake, and Cloudera, to Databricks-based architectures. Data Ingestion and Processing: Establish scalable patterns for batch and real-time data processing using Structured Streaming, Delta Live Tables (DLT), Auto Loader, and Apache Spark optimization techniques. Technical Leadership and Quality Assurance: Guide engineering teams through complex delivery initiatives, establish reference architectures, conduct architectural reviews, review code contributions, and implement standardized CI/CD practices using GitHub Actions, GitLab, Azure DevOps, Terraform, and Databricks Asset Bundles (DABs). Performance Optimization: Identify and resolve complex Spark and Delta Lake performance challenges by optimizing query execution, partitioning, Z-Ordering, caching, and Photon configurations. Cloud Cost Management: Establish FinOps practices and compute governance standards, including cluster policies, autoscaling, auto-termination, serverless configurations, and cost allocation models to optimize Databricks consumption. AI and Advanced Analytics Integration: Collaborate with machine learning and AI specialists to integrate MLflow, Feature Store, Model Serving, Vector Search, and Retrieval-Augmented Generation (RAG) pipelines into enterprise data platforms. Client Advisory and Stakeholder Engagement: Serve as a trusted technical advisor to senior client stakeholders, including Chief Architects, Directors of Engineering, and Chief Data Officers. Lead technical workshops, architecture design sessions, and proof-of-concept evaluations. Technology Partner Collaboration: Work closely with Databricks Partner Solutions Architects and field engineering teams to assess emerging capabilities, evaluate preview features, and ensure alignment with platform best practices. Architectural Innovation and Knowledge Sharing: Develop reusable architectural blueprints, migration accelerators, reference architectures, and standardized configurations to strengthen internal technical capabilities and delivery practices. Requirements: Professional Experience: At least 8 years of experience in data architecture, data engineering, or enterprise software development, including a minimum of 3 years designing and architecting enterprise solutions based on Apache Spark and the Databricks Data Intelligence Platform. Databricks Certification: Databricks Certified Data Engineer Professional or Databricks Certified Machine Learning Professional certification. Cloud Architecture Expertise: Strong architectural proficiency in at least one major cloud platform (AWS, Azure, or GCP), including VPC/VNet networking, private endpoints, cloud identity and access management, Entra ID integration, and cloud object storage solutions such as S3, ADLS Gen2, or GCS. Data Modeling and Governance: Advanced knowledge of modern data modeling methodologies, including dimensional modeling, Data Vault, and Medallion Architecture, with hands-on experience implementing Unity Catalog across multiple workspaces in production environments. Programming and Automation: Advanced proficiency in Python, PySpark, and SQL, combined with practical experience using Infrastructure as Code tools such as Terraform and orchestration platforms such as Databricks Workflows or Airflow. Technical Delivery Leadership: Demonstrated experience leading technical delivery teams, translating complex business and functional requirements into architectural designs, technical specifications, and user stories, and resolving critical implementation challenges. DevOps and Engineering Practices: Strong understanding of CI/CD pipelines, code quality standards, infrastructure automation, and modern software engineering practices. Client Communication and Advisory Skills: Excellent communication, presentation, and stakeholder management abilities, with the confidence to facilitate technical discussions and advise senior decision-makers. Additional Technical Experience: Familiarity with modern data ecosystem integrations, including Power BI, Tableau, reverse ETL, dbt, and data observability platforms, is highly desirable. Security and Compliance: Experience designing and implementing solutions aligned with enterprise regulatory and security frameworks such as HIPAA, HITRUST, SOC 2, or FedRAMP. Security Clearance: An active Public Trust or Secret clearance, or eligibility to obtain one, is considered an advantage, particularly for regulated and public-sector engagements. Benefits: Competitive Compensation: Estimated annual salary range of $140,000–$210,000, depending on experience, technical expertise, qualifications, certifications, and other relevant factors. Remote Work: Fully remote working arrangement within the United States. Technical Leadership Opportunities: Opportunity to take ownership of complex enterprise architecture initiatives and influence large-scale data modernization projects. Professional Development: Exposure to advanced cloud technologies, the Databricks ecosystem, artificial intelligence, machine learning, and emerging data engineering capabilities. Industry Exposure: Opportunity to collaborate with enterprise clients, senior technology leaders, and specialized engineering teams across complex technical environments. Innovation and Knowledge Sharing: Contribute to reusable technical solutions, architectural frameworks, and best practices that support continuous improvement and innovation.