Senior Data Engineer

Accountabilities:: Streaming Data Pipeline Development: Design, build, and maintain robust Apache Spark Structured Streaming applications that process Change Data Capture (CDC) events into reliable, near-real-time data models. Stream Processing Reliability: Implement and optimize stateful processing, slowly changing dimensions (SCD), checkpointing, state stores, late-arriving and partial records, and safe recovery mechanisms to ensure accurate and consistent data processing. AWS Infrastructure Management: Own and scale AWS EMR clusters, managing instance sizing, resource allocation, managed scaling, multi-cluster deployments, driver and executor memory optimization, and workload consolidation to improve efficiency and reduce infrastructure costs. Lakehouse Management: Maintain and optimize Apache Iceberg tables at scale by designing partitioning strategies and implementing compaction, snapshot expiration, and delete-file cleanup, including specialized maintenance for high-churn tables. Data Source Integration: Onboard new business domains onto the streaming platform by configuring CDC sources and destinations through Estuary Flow, setting up Kafka-compatible topics, validating backfills in Snowflake, and coordinating with source database owners. Production Troubleshooting: Investigate and resolve complex production incidents across the data platform, addressing issues such as connection failures, offset regressions, token expiration, schema inconsistencies, and write conflicts while focusing on root-cause resolution. Monitoring and Observability: Develop monitoring capabilities and diagnostic tools to improve incident detection and resolution, including integrating Spark driver and executor logs with Grafana Loki, implementing watchdog mechanisms, and creating AI-assisted operational runbooks. CI/CD and Developer Experience: Improve deployment workflows, configuration management, automated pipelines, and operational safeguards to enable engineering teams to independently and safely deploy new data models. Cross-Team Collaboration: Support partner engineering teams with ad hoc data ingestion, backfill requests, technical escalations, and platform-related guidance. Requirements: At least 7 years of professional experience in data engineering , including a minimum of 3 years operating production-grade streaming data pipelines. Apache Spark Expertise: Strong hands-on experience with Apache Spark, particularly Structured Streaming, state stores such as RocksDB, checkpointing, stateful processing, and performance optimization. Lakehouse Technologies: Production experience with modern lakehouse table formats, ideally Apache Iceberg, including partitioning strategies, table maintenance, compaction, and snapshot management. Experience with Delta Lake or Apache Hudi is also relevant. Cloud Infrastructure: Experience operating Spark workloads on AWS EMR, Amazon EKS, or comparable platforms, with a strong understanding of cluster management, capacity planning, autoscaling, resource allocation, and infrastructure cost optimization. Streaming and CDC: Solid understanding of Change Data Capture principles and Kafka semantics, including offsets, consumer configurations, message delivery guarantees, and event processing reliability. Programming Skills: Strong proficiency in Python and SQL. Experience with Scala or Java is considered an advantage. Orchestration and Automation: Experience with Apache Airflow or similar workflow orchestration tools, alongside CI/CD pipelines and infrastructure-as-code practices. Troubleshooting and Analytical Skills: A systematic approach to debugging, with the ability to interpret logs, investigate complex issues, formulate hypotheses, and validate solutions using data. Communication and Collaboration: Ability to work independently while collaborating effectively with distributed engineering teams and cross-functional stakeholders. Nice to Have: Experience with Snowflake, particularly querying external or Iceberg tables, as well as monitoring and observability tools such as Grafana, Loki, or Alloy. Platform Enablement: Previous experience building self-service data platforms or enabling other engineering teams to independently manage data workflows is an advantage. Containerization: Familiarity with Kubernetes (K8s) is a plus. Benefits: Competitive Compensation: Annual gross base salary ranging from $50,000 to $80,000 USD , depending on experience, technical skills, and location. Localized Compensation: Salary offers are adjusted to the candidate's country and paid in the relevant local currency using market-aligned conversion rates. Equity Opportunities: Stock grant opportunities, depending on role, employment status, and location. Flexible Working Arrangements: Fully remote work, with optional flexible office memberships through IWG. Additional Benefits: Access to additional perks and benefits based on employment status and country of residence. International Exposure: Opportunity to collaborate with globally distributed engineering teams and contribute to a large-scale international technology platform. Technical Growth: Work with modern data engineering technologies, including Spark Structured Streaming, Apache Iceberg, AWS, CDC, and AI-assisted operational tools. Meaningful Technical Ownership: Take responsibility for the architecture, reliability, scalability, and continuous improvement of a business-critical data platform. Inclusive Workplace: Join an environment committed to equal opportunities, diversity, accessibility, and professional development.