DE&A - Senior Data Engineer (CDAI)

Core Skills & Experience • Strong hands-on experience designing, building, and optimizing enterprise-scale data pipelines, data architecture, and data products for performance, scalability, and efficient AI agent consumption. • Deep expertise in Starburst/Trino and distributed SQL, including query optimization, materialized views (MVs), data modeling, storage design, and tuning data products for low-latency, high-concurrency AI agent access. • Strong experience optimizing data storage, partitioning, indexing, file formats, and materialization strategies to improve query performance, data freshness, and cost efficiency. • Hands-on development experience with Apache Spark using Scala and/or Python/PySpark, including migrating complex, SQL-intensive transformations to distributed processing frameworks to improve data curation speed and efficiency. • Strong ability to evaluate and redesign data processing workflows, selecting the appropriate technologies and execution patterns across SQL, Spark, and other processing frameworks to optimize performance. • Experience with RDBMS technologies and modern lakehouse architecture, preferably Apache Iceberg, including table design, partitioning, compaction, and data lifecycle optimization. • Understanding of integration patterns across NiFi, Kafka, Spark, RDBMS, and lakehouse platforms for efficient batch and real-time data processing. • Strong software engineering practices, including CI/CD, automated testing, source control, infrastructure automation, observability, and production support. • Experience with data governance, metadata management, lineage, data quality, and secure data access in enterprise environments. NA