Senior Solution Architect- Spark & Databricks Expert

Accountabilities: Lead customer engagements from initial discovery and technical assessment through solution architecture, implementation, and final delivery, taking full ownership of outcomes. Design, architect, implement, and optimize large-scale Apache Spark and Databricks workloads, ensuring scalability, reliability, and cost efficiency. Apply advanced knowledge of Spark internals to troubleshoot, debug, and optimize distributed data processing jobs and SQL workloads. Develop and deliver proofs of concept (PoCs), technical demonstrations, and customer workshops to communicate solution capabilities and validate architectural approaches. Act as a trusted technical advisor, proactively identifying customer challenges, anticipating potential issues, and recommending innovative, data-driven solutions. Drive technical decision-making and translate complex business requirements into practical, scalable data architectures. Collaborate with product and engineering teams to enhance platform capabilities, improve technical solutions, and strengthen the overall customer experience. Establish and promote engineering best practices across PySpark, Databricks, and Lakehouse architecture. Mentor junior engineers and support the development of technical expertise across data engineering teams. Continuously identify opportunities to improve data processing performance, platform governance, and overall solution efficiency. Requirements: Between 5 and 10+ years of professional experience in data engineering, analytics, or enterprise data solution delivery. At least 3 years of hands-on experience with Databricks, including workload optimization, debugging, performance tuning, and governance. Expert-level understanding of Apache Spark fundamentals and internals, with strong practical experience in PySpark development. Proven expertise in Databricks SQL (DBSQL), including query optimization and performance tuning. Strong knowledge of modern Lakehouse architectures, Delta Lake, Structured Streaming, and cloud infrastructure. Demonstrated ability to architect and deliver complex, large-scale data solutions independently, with minimal supervision. Strong analytical and problem-solving skills, with the ability to take initiative, make technical decisions, and manage complex challenges autonomously. Excellent communication and stakeholder management skills, particularly the ability to explain sophisticated technical concepts and architectural decisions to business leaders and non-technical audiences. Ability to lead customer-facing technical engagements and build trusted relationships through proactive collaboration and solution-oriented thinking. Preferred: Databricks Certified Data Engineer Professional certification. Experience with advanced Databricks capabilities, including Delta Live Tables (DLT) and Unity Catalog (UC). Previous consulting experience involving large-scale data migrations, enterprise analytics transformations, or AI-assisted analytics initiatives is considered an advantage. Benefits: Fully remote working arrangement within the United States. Full-time contract opportunity. Opportunity to work on complex, enterprise-scale data engineering and analytics projects. Exposure to advanced technologies, including Apache Spark, Databricks, PySpark, Delta Lake, and modern Lakehouse architectures. Opportunity to lead strategic customer engagements and influence enterprise data solutions. Collaborative environment with opportunities to work alongside product, engineering, and customer-facing teams. Professional growth through hands-on experience with advanced data platforms, architecture, and cloud technologies. Opportunity to mentor other engineers and contribute to technical best practices.