ML Platform Engineer, Distributed Training

Accountabilities: Optimize the training platform: Partner with infrastructure teams to improve GKE-based training environments, including GPU node pools, job scheduling, queueing, resource quotas, and capacity management. Diagnose platform failures: Investigate underlying causes of infrastructure and training issues, resolve systemic problems, and implement durable fixes rather than relying on temporary workarounds. Tune GPU networking: Optimize multi-node GPU communication and training throughput using technologies and techniques such as RDMA, GPUDirect, NCCL, and GPU interconnect topology. Run distributed training workloads: Execute, scale, profile, and debug multi-node training jobs using PyTorch-based frameworks such as FSDP, DeepSpeed, Megatron, or NeMo, alongside Ray Train. Identify performance bottlenecks: Analyze compute utilization, memory consumption, input pipelines, and network communication to improve training efficiency and scalability. Design performance benchmarks: Develop and run benchmarks that measure throughput, scaling efficiency, and cost per training run, using the results to guide engineering decisions. Build model evaluation systems: Create and maintain evaluation harnesses that objectively measure the impact of changes to training recipes, models, and platform configurations. Improve observability and testing: Instrument training workloads and platform components with telemetry and monitoring, and maintain end-to-end pipeline tests that reflect real user workflows. Own technical outcomes: Proactively identify gaps, communicate project status and risks, propose solutions, and support the scientists who rely on the platform. Collaborate and communicate: Explain and defend architectural decisions to engineers and researchers, work effectively across teams, and adapt to evolving priorities and organizational structures. Requirements: Professional experience: At least 5 years of engineering experience, including recent hands-on work with GPU training infrastructure and large-scale model training. Python engineering: Strong production-level Python skills, including writing clean, maintainable code, automated tests, participating in code reviews, and following sound software development practices. Kubernetes and GKE: Practical experience with Kubernetes on Google Cloud, including GPU node pools, job scheduling and queueing systems such as Kueue, and resource quota and capacity management. Distributed training expertise: Direct experience running and debugging multi-node PyTorch training jobs and working with Ray Train. GPU networking and interconnects: Strong understanding of RDMA, GPUDirect, NCCL tuning, and multi-node GPU architectures involving H200 or GB200 hardware. Performance analysis: Demonstrated ability to design and execute training benchmarks, evaluate model performance, identify bottlenecks, and use empirical evidence to guide optimization. Technical depth: Ability to explain system architecture, implementation choices, and performance behavior in detail, demonstrating independent engineering judgment rather than relying on AI coding tools to supply the underlying reasoning. Ownership and communication: A track record of accountability, proactive problem-solving, transparent communication, and effective collaboration in demanding engineering environments. Preferred domain experience: Familiarity with healthcare or life sciences data, including genomics, clinical, or biomedical information, and experience working under HIPAA requirements. Additional preferred skills: Experience building observability and telemetry for training platforms, including metrics, tracing, alerting, and end-to-end pipeline testing. Benefits: Annual salary: $159,300–$219,245, with the final salary range determined by the assigned level and applicable factors such as experience, expertise, interview performance, and geographic location. Comprehensive health insurance: Healthcare coverage to support employee well-being. Paid time off: Vacation and general paid leave. Paid holidays: Paid time off for designated holidays. Sick leave: Leave to support health and recovery. Parental leave: Leave to support employees and their families. Bereavement leave: Support during times of loss. Retirement benefits: 401(k) plan with an employer match. Employee referral bonuses: Opportunities to receive bonuses for eligible referrals. Employment type: The position may be offered as full-time employment or contract work, as indicated in the original listing.