Staff Software Engineer, Tech Foundations

Accountabilities: Technical Leadership & Architecture: Define the technical vision and architecture for major areas of the Observability platform, leading initiatives from initial design and prototyping through production deployment and ongoing operations. Distributed Systems Development: Design, build, and maintain highly scalable and reliable distributed systems capable of processing tens of millions of events per second. Metrics Infrastructure: Scale and optimize the metrics pipeline using technologies such as OpenTelemetry, vmagent, and Grafana Mimir, while managing infrastructure costs through aggregation, sampling, and unused-metric filtering. Logging & Tracing Modernization: Lead the transition from legacy log aggregation systems to OpenTelemetry-based pipelines and improve log and trace storage capabilities using technologies such as OpenSearch. Query Infrastructure: Develop fast, reliable, and secure query systems for engineers and AI agents, incorporating rate limiting, access controls, isolation, and caching to protect critical incident-response workflows. Telemetry Cost Optimization: Build self-service tools that provide engineering teams with visibility into telemetry costs, including cost attribution by team and service, as well as controls for data granularity, sampling, and retention. Observability & Reliability: Improve incident detection through anomaly detection, automated signal correlation, and enhanced monitoring capabilities. Cross-functional Collaboration: Align technical roadmaps with infrastructure and product teams, influence build-versus-buy decisions, and communicate architectural tradeoffs and investment priorities to engineering leadership. Engineering Excellence: Write high-quality, maintainable, and well-tested code, participate in design reviews, improve operational readiness, and contribute to relevant open-source projects. Mentorship & Knowledge Sharing: Mentor engineers, strengthen technical practices across the team, and help developers instrument services and independently investigate production issues. Requirements Professional Experience: At least 9 years of software engineering experience, with a proven track record of designing, building, and operating large-scale distributed systems in production environments. Technical Leadership: Demonstrated experience leading the technical direction of complex systems or multi-team initiatives, including architecture design, stakeholder alignment, delivery, and production operations. Observability Expertise: Deep experience in at least one major telemetry domain at scale, including metrics platforms such as Prometheus, Grafana Mimir, or VictoriaMetrics; logging technologies such as OpenSearch or Kafka; or distributed tracing solutions such as OpenTelemetry, Jaeger, or ClickHouse. Programming Skills: Strong software development capabilities in Go or Java/Kotlin, with the ability to navigate unfamiliar codebases and contribute to upstream open-source projects. Cloud Infrastructure: Hands-on experience with cloud-native technologies, particularly Kubernetes, AWS, and service mesh solutions such as Istio or Envoy. Communication & Collaboration: Excellent written and verbal communication skills, including the ability to produce clear technical design documents, explain architectural tradeoffs, and present investment recommendations to engineering leadership. Problem-solving: Strong analytical and architectural thinking, with the ability to address complex scalability, reliability, performance, and cost challenges. Open-source Contributions (Preferred): Experience contributing to OpenTelemetry, Prometheus, Grafana Mimir, VictoriaMetrics, OpenSearch, Jaeger, or comparable projects. Advanced Technical Knowledge (Preferred): Familiarity with time-series storage internals, high-cardinality datasets, query optimization, anomaly detection, automated correlation, or SLO-based alerting. Additional Expertise (Preferred): Experience with telemetry cost attribution, large-scale infrastructure cost optimization, AI-agent-facing APIs or tools (including MCP servers), and columnar analytics platforms such as ClickHouse. Benefits Competitive annual base salary ranging from $212,000 to $265,000 USD . Potential eligibility for performance-based bonuses. Equity compensation opportunities. Comprehensive employee benefits package. Employee Travel Credits. Remote-first working arrangement for eligible US locations. Flexibility to work remotely, with occasional office visits or team offsites as agreed with management. Opportunities to contribute to large-scale engineering initiatives and open-source projects. Collaborative environment focused on technical innovation, professional development, and knowledge sharing. Inclusive workplace culture that values diverse perspectives and equal employment opportunities.