Senior Software Engineer, SRE and Production Engineering - DGX Cloud

Accountabilities: You will design, build, and maintain the software and operational tooling required to deploy, manage, and support large-scale GPU infrastructure. Your main responsibilities will include: Develop automation for bare-metal server provisioning, hardware validation, firmware and software upgrades, repairs, and cluster lifecycle management. Build tools that interact with Baseboard Management Controller (BMC) and Redfish interfaces to monitor hardware health, manage server states, and support recovery workflows. Operate and advance NVIDIA NVL72 systems and BlueField-3 or newer Data Processing Units (DPUs) across cloud partner and on-premises environments. Diagnose and resolve complex failures involving servers, DPUs, GPU and CPU systems, networking, Linux, and Kubernetes. Convert recurring infrastructure issues into automated detection, troubleshooting, and repair processes. Define validation procedures and production handoff criteria to ensure new infrastructure capacity is introduced safely and consistently. Participate in on-call rotations, incident response, root-cause investigations, and follow-up activities to implement permanent corrective solutions. Collaborate with hardware, networking, platform engineering, data center operations, and partner teams to resolve issues across organizational boundaries. Improve infrastructure reliability, operational consistency, and automation across the production fleet. Requirements The ideal candidate brings extensive software engineering and production infrastructure experience, with strong hands-on expertise in bare-metal systems and large-scale hardware operations. The following qualifications and capabilities are expected: At least 12 years of experience building software for or operating production infrastructure, including substantial hands-on bare-metal experience. Strong programming skills in Go or Python, with a proven track record of delivering production automation and services. Direct experience using BMC and Redfish interfaces for server provisioning, hardware health inspection, power management, or fault diagnosis. Practical experience working directly with NVIDIA GPU hardware, particularly NVL72 systems, and BlueField-3 or newer DPUs. Strong knowledge of Linux, firmware and driver management, network boot processes, and server lifecycle operations from initial provisioning through repair. Experience maintaining production reliability through on-call responsibilities, incident response, observability, and implementation of durable solutions. Ability to troubleshoot complex issues across hardware, host operating systems, networking, and distributed services. Strong communication skills and demonstrated ownership of technical problems involving multiple teams. Bachelor's or Master's degree in Computer Science, or equivalent practical experience. Additional experience that would be an advantage: Experience operating BlueField DPUs in DPU mode, including host-to-DPU connectivity and lifecycle troubleshooting. Familiarity with NVLink, InfiniBand, Spectrum-X, or GPU cluster performance validation. Experience designing safe and repeatable workflows for rack-scale deployment, firmware upgrades, hardware replacement, and customer handoffs. Knowledge of Kubernetes, GitOps, Argo CD, Service Level Objectives (SLOs), and fleet-wide automation. Benefits Fully remote work arrangement in Canada. Full-time position. Competitive compensation package with eligibility for equity. Access to employee benefits and wellness programs. Opportunity to work hands-on with advanced GPU infrastructure and large-scale AI workloads. Exposure to complex production engineering challenges across cloud and on-premises environments. Collaboration with specialized teams working across hardware, networking, platform engineering, and data center operations. Opportunity to contribute to the development and operation of next-generation AI infrastructure.