Senior Data & Integration Engineer – Incident Management Platform

Accountabilities:: Design, develop, and maintain production-grade API and webhook integrations connecting alerting platforms, Jira Cloud, Jira Service Management (JSM), and internal services with centralized data infrastructure. Build and implement AI-agent integrations within Jira workflows, including tool calling, structured outputs, auditable write-backs, and human-in-the-loop approval mechanisms. Develop reliable and scalable data pipelines using Databricks, Delta Lake, PySpark, and SQL to ingest and process data from AWS S3 and other integrated systems. Ensure data pipeline reliability through incremental ingestion, idempotent processing, schema evolution, and data reconciliation mechanisms. Design and maintain governed incident data models and curated datasets supporting operational and SRE performance metrics, including Mean Time to Detect (MTTD), Mean Time to Resolve (MTTR), incident coverage, and recurring incident analysis. Establish and maintain data governance practices, including documented data lineage, access controls, data retention policies, and appropriate security measures. Develop comprehensive testing strategies and automated test suites to validate integrations, data pipelines, and AI-powered workflows. Implement failure recovery mechanisms, data backfills, and reconciliation procedures to ensure platform resilience and data consistency. Create clear operational documentation, technical specifications, and runbooks to support ongoing maintenance and knowledge transfer. Take end-to-end ownership of technical deliverables, ensuring successful implementation, testing, deployment, and handover to internal operations teams. Collaborate with engineering, operations, and other technical stakeholders to ensure the platform meets business and operational requirements. Contribute to continuous improvement initiatives focused on system reliability, automation, scalability, and engineering efficiency. Requirements: At least 5 years of professional experience in Data Engineering, Backend Integration Engineering, or a closely related technical role. Proven experience designing, developing, and maintaining production-grade data pipelines, backend services, and system integrations. Strong hands-on expertise with Databricks, Delta Lake, and PySpark or Spark SQL in production environments. Experience working with Databricks jobs, workflows, medallion architecture, and incremental data ingestion strategies. Demonstrated ability to build and maintain REST APIs, webhooks, and lightweight backend services using Python frameworks such as FastAPI or Flask. Strong understanding of API authentication, retry mechanisms, rate limiting, error handling, and idempotency. Practical experience with AWS services, particularly Amazon S3 and IAM, for secure data storage, access management, and ingestion workflows. Hands-on experience integrating Large Language Models (LLMs) or AI agents into real-world business workflows. Understanding of agentic workflow implementation, including tool calling, human-in-the-loop validation, structured outputs, and comprehensive audit logging. Strong software engineering fundamentals, including Git version control, CI/CD automation, and automated unit and integration testing. Ability to develop clear technical documentation, operational procedures, and system recovery runbooks. Advanced professional English proficiency, both written and spoken, with the ability to communicate effectively across multicultural and globally distributed teams. Strong analytical thinking, problem-solving abilities, technical ownership, and attention to detail. Ability to work independently, manage priorities, and collaborate effectively in a remote engineering environment. Nice to have: Direct experience integrating Jira Cloud and Jira Service Management through APIs and webhooks. Familiarity with data governance and security tools, particularly Databricks Unity Catalog, row-level and column-level security, data masking, and Personally Identifiable Information (PII) protection. Understanding of Site Reliability Engineering (SRE) practices and incident response metrics, including MTTD and MTTR. Experience with observability and monitoring platforms such as AWS CloudWatch, Datadog, or New Relic. Previous experience in fintech, payment platforms, or highly regulated data environments. Benefits: Work arrangement: 100% remote position. Contract duration: Initial 3-month contract, with the possibility of extension to 6 months. Strategic impact: Opportunity to contribute to a high-visibility engineering initiative involving the development of a centralized incident management platform. Technical environment: Hands-on exposure to modern technologies, including Databricks, Delta Lake, PySpark, AWS, APIs, and AI-powered automation. International collaboration: Work with a multicultural, globally distributed team and experienced technical leaders. Professional development: Opportunities to expand technical expertise and contribute to complex, high-impact engineering projects. Growth opportunities: Potential to develop professionally through challenging technical responsibilities and exposure to modern engineering practices. Collaborative culture: Join an inclusive, remote-first environment that values technical excellence, autonomy, innovation, and knowledge sharing.