Senior Software Engineer, Datacenter Automation Platforms

Collaborate with NVIDIA professionals responsible for building the platforms that operate AI factories at scale. We drive product roadmaps, architecture, multi-functional alignment, and implementation across multiple open source and internal-facing products. We partner with executive leadership to align platform strategy and drive adoption of innovative approaches to software development across engineering teams. Our team mentors engineering talent and turns successful domain solutions into reusable platform capabilities used across the organization. What you'll be doing: Build self-service SDKs, workflow templates, reference implementations, and a catalog of reusable agents, tools, and connectors. Prioritize automation by customer impact, frequency, operational risk, and encouraged reduction in human effort. Translate operational intent into versioned, testable policies that define triggers, prerequisites, permitted actions, approval requirements, maintenance windows, capacity limits, and recovery or blocking issue paths. Define common contracts for tools, state, events, and workflow execution. Establish versioning and compatibility standards, guide migration from fragmented implementations to shared services, and select the runtime and durable execution architecture. Build agents that correlate hardware diagnostics, logs, metrics, topology, configuration, and incident history to investigate failures, propose evidence-backed diagnoses, and produce executable remediation plans. Enforce authorization independently of model output, retain action histories, and support human review, bounded retries, cancellation, and compensating actions. Automate provisioning validation, node quarantine, workload-aware maintenance, repair, upgrades, and return to service. Integrate with Kubernetes, fleet management systems, and relevant schedulers and hardware interfaces. Tackle fleet-scale problems including partial failures, duplicate events, stale state, conflicting remediations, noisy telemetry, control-plane bottlenecks, and correlated rack or cluster failures. Establish shared evaluation, CI/CD, staged rollout, and observability capabilities across teams. Drive cross-team adoption through build reviews, migration plans, documentation, and hands-on partnership. Align infrastructure, networking, storage, security, and workload teams on shared priorities and contribution standards. What we need to see: 8+ years of experience in software engineering or a related field. BS in Computer Science, Engineering, Physics, Mathematics, or a comparable field, or equivalent experience. Expert level knowledge of a systems programming language (Go, Python, and/or Rust) and a solid understanding of Data Structures and Algorithms. Strong background in Linux system administration and management. Experience handling cluster orchestration tools, including Kubernetes and/or SLURM. Understanding of software development patterns dedicated to agents and current security practices for agentic systems. Understanding of performance, security, and reliability in complex distributed systems, including system-level architecture, data synchronization, fault tolerance, and state management. Ways to stand out from the crowd: Experience with open source software projects, licensing, and governance. Proficiency in understanding and managing large-scale distributed systems independently of cloud providers. Deep knowledge of datacenter operations and GPU hardware. Advanced hands-on experience with cluster management systems (Kubernetes, SLURM). Hands-on experience in Machine Learning Operations, Training, and/or Security fields. #LI-Hybrid Your base salary will be determined based on your location, experience, and other objective job-related factors. The expected base salary ranges for this position are: 184,000 USD – 287,500 USD for Level 4; and 224,000 USD – 356,500 USD for Level 5. You will also be eligible for equity and benefits . Applications for this job will be accepted at least until November 14, 2026. This posting is for an existing vacancy. NVIDIA uses AI tools in its recruiting processes. NVIDIA is committed to fostering an inclusive work environment and proud to be an equal opportunity employer. As we highly value diversity in our current and future employees, we do not discriminate (including in our hiring and promotion practices) on the basis of race, religion, color, national origin, gender, gender expression, sexual orientation, age, marital status, veteran status, disability status or any other characteristic protected by law.