Senior Network Engineer - InfiniBand / UFM
Lightning AI
Tailor my CV for this job, freeView job and applyYour CV rewritten for this role, from your real experience. Sign in with Google, nothing to install.
Got this interview? Our apps help you get the job.
Skills named in this job
Read from the description itself, not inferred.
This role on the market
8 open infiniband roles across 2 companies are on ApplySarthi right now, most of them in Bengaluru (2).
What infiniband roles keep asking for: Ansible (38%), Linux (38%), Python (38%), CI/CD (25%), Deep learning (25%), System design (25%), iOS (25%), AWS (12%) — counted across their open postings here.
Network Engineer jobs in the United States · Remote Network Engineer jobs · Ansible jobs · Linux jobs · Observability jobs · PyTorch jobs
Lightning AI has 53 open roles listed here.
- Senior Product Manager, AI Infrastructure
- Financial Analyst
- Senior Software Engineer
- Senior Software Engineer
- Technical Program Manager, Infrastructure Delivery
Counted across 14 company job boards, updated as roles open and close.
Preparing for this interview
Interviews for infiniband roles keep coming back to Ansible, Linux, Python, CI/CD. Practise those questions before you sit with Lightning AI.
Questions you are likely to be asked
- Why do you want to join Lightning AI?
- What is your experience with Observability? Tell me one thing you learned the hard way.
- How do you keep secrets and access safe in your infrastructure?
- Walk me through how code gets from a commit to production where you work.
- Tell me about an outage you handled. What did you learn from it?
Prep Sarthi gives you a free mock interview: an AI interviewer asks you questions like these out loud, from your own CV and this job, and shows your score and your weakest answer.
Practise the Senior Network Engineer - InfiniBand / UFM at Lightning AI interview free →Who We Are
Lightning AI is the company behind PyTorch Lightning. Founded in 2019, we build an end-to-end platform for developing, training, and deploying AI systems—designed to take ideas from research to production with less friction.
Through our merger with Voltage Park, a neocloud and AI Factory, Lightning AI combines developer-first software with cost-efficient, large-scale compute. Teams get the tools they need for experimentation, training, and production inference, with security, observability, and control built in.
We serve solo researchers, startups, and large enterprises. Lightning AI operates globally with offices in New York City, San Francisco, Seattle, and London, and is backed by Coatue, Index Ventures, Bain Capital Ventures, and Firstminute.
The Way We Work
The people who thrive here are builders who move fast, communicate openly, take ownership, and continuously improve themselves, their teams, and our company. Here's what that looks like in practice:
- Move with Urgency: We move quickly, make thoughtful decisions, and keep momentum. We value action over perfection and learn by shipping.
- Take Ownership: We own outcomes, not just our individual work. We make decisions that move the company forward and follow through.
- Communicate Openly: We communicate directly, seek to understand, and create clarity for others. Honest conversations help us move faster together.
- Build Great Teams: We lead by example, empower others, and create healthy teams where people can do their best work.
- Raise the Bar: We're always improving ourselves. We learn from feedback, consistently challenge ourselves to grow, and focus on the work that matters most.
- Think Long-Term: We design for what's next. We create scalable systems, simplify complexity, and use AI and automation to amplify our impact.
What We Are Looking For
We are seeking an experienced Senior Network Engineer (InfiniBand / UFM) to design, deploy, automate, and operate next-generation AI Factory networking infrastructure supporting large-scale GPU clusters. This role is responsible for building and maintaining high-performance NVIDIA Quantum InfiniBand fabrics that power AI training and inference environments utilizing NVIDIA UFM, NCCL, RoCEv2, and modern data center technologies.
The ideal candidate has deep expertise in InfiniBand networking, NVIDIA Unified Fabric Manager (UFM), large-scale GPU deployments, Linux networking, automation, and troubleshooting distributed AI workloads. The network is the foundation of distributed AI training. Performance bottlenecks at the fabric layer directly impact customer workloads. This role will shape how Lightning scales from tens of thousands of GPUs to significantly beyond — ensuring deterministic performance, predictable scaling, and enterprise-grade reliability. If you want to architect infrastructure that powers frontier AI research, this is the role for you.
This role is hybrid with a minimum of 2 in-office days per week in San Francisco, Seattle, or NYC, with fully remote work considered for candidates outside of our office hub locations. All employees participate in occasional team and company offsites.
What You’ll Do
- Design, deploy, and maintain large-scale NVIDIA InfiniBand fabrics supporting AI/ML GPU clusters.
- Deploy and administer NVIDIA Unified Fabric Manager (UFM) Enterprise for monitoring, provisioning, telemetry, and fabric health.
- Configure and optimize NVIDIA Quantum and Quantum-2 InfiniBand switches.
- Troubleshoot fabric performance issues impacting NCCL, MPI, GPUDirect RDMA, and AI training jobs.
- Implement and validate fat-tree, Dragonfly+, Clos, and spine-leaf network architectures.
- Perform firmware lifecycle management for InfiniBand switches, adapters (HCAs), and UFM infrastructure.
- Optimize congestion control, adaptive routing, QoS, and traffic engineering for high-performance GPU communication.
- Work closely with AI platform, GPU infrastructure, storage, and systems engineering teams to deploy scalable AI Factory environments.
- Automate network provisioning using Python, Ansible, Git, REST APIs, and Infrastructure-as-Code methodologies.
- Monitor network health using UFM telemetry, Prometheus, Grafana, and other observability platforms.
- Support high availability, maintenance windows, incident response, root cause analysis, and capacity planning.
- Participate in architecture reviews and define networking standards for AI infrastructure.
What You'll Need
Required Qualifications
- 7+ years of experience in large-scale data center networking.
- Experience deploying, operating, and troubleshooting InfiniBand environments.
- Strong understanding of Layer 2 and Layer 3 networking, including BGP, EVPN, VXLAN, and spine-leaf architectures.
- Strong Linux administration experience.
- Experience with network automation using Python, Ansible, Terraform, or similar tools.
- Experience with network troubleshooting, observability, and telemetry tooling.
- Excellent troubleshooting, documentation, and communication skills.
Ideal Experience
- Experience with Netris, Terraform, or similar infrastructure tooling.
- Experience with multi-region backbone design.
- Exposure to bare-metal provisioning systems.
- Experience operating HPC, GPU-dense, or other large-scale infrastructure environments.
- Experience working in high-growth infrastructure startups
Compensation
We are committed to offering competitive compensation that reflects the value each team member brings to our mission. Final offers are based on factors such as experience, skills, geographic location, and role expectations. In addition to base salary, our total rewards package for eligible roles includes a discretionary bonus, a meaningful equity component, and comprehensive benefits.
Benefits and Perks
We offer a comprehensive and competitive benefits package designed to support our employees’ health, well-being, and long-term success:
- Comprehensive Health Coverage: Medical, dental, and vision coverage for employees and eligible dependents.
- Meaningful Equity: RSUs that give employees a stake in the company's long-term success.
- Retirement Savings: 401(k) matching (U.S.) and pension contributions (U.K.).
- Flexible Time Off: Unlimited PTO, company holidays, and floating holidays to support work-life balance.
- Company-Wide Winter Break: Two weeks of company closure each winter to disconnect and recharge.
- Paid Parental & Family Leave: Paid leave to support you and your family through life's important moments.
- Professional Development: Annual learning and development allowance to support your professional growth.
- Wellness Benefits: Wellness and work-from-home stipends to support your physical and mental well-being.
- Sabbatical Program: Four weeks of paid sabbatical leave after four years of service.
- Flexible Work: Flexible schedules and a hybrid work model for our office-based teams.
- In-Office Meals: Complimentary meals at our office hubs.
Benefits may vary by location, team, and role.
At Lightning AI, we are committed to fostering an inclusive and diverse workplace. We believe that diverse teams drive innovation and create better products. We provide equal employment opportunities to all employees and applicants without regard to race, color, religion, gender, sexual orientation, gender identity, national origin, age, disability, veteran status, or any other protected characteristic. We are dedicated to building a culture where everyone can thrive and contribute to their fullest potential.
Match this job to your CV
ApplySarthi scores your CV against this role, shows the skills you are missing, and writes a tailored version for the application.
Check my match →Similar open roles
- Not seeing the right opportunity right now?Lightning AI
- Research Engineer, AI/ML SystemsLightning AI
- Senior Network EngineerLightning AI
- Senior Research Engineer, LLM Training & Post-TrainingLightning AI
- Senior Technical Program ManagerLightning AI
- Technical Program Manager, Infrastructure DeliveryLightning AI
- Infrastructure Engineer (GPU & Compute)Lightning AI
- Infrastructure Engineer (Storage)Lightning AI
Need answers during your interview? Try Live Sarthi.
Live Sarthi, an Interview Sarthi app, shows answer suggestions during the call.
- Hidden from supported screen sharingThe overlay stays out of supported Windows screen captures.
- Answers start in about 1.5 secondsResponse time varies with your connection and model.
- From your own CVYour projects and your experience, not a generic script.
- 30 minutes freeThen ₹99 for a 2-day pass with unlimited calls — you pay for the days you are interviewing, not a subscription.
A Windows app, from the same team as ApplySarthi.
Listed on greenhouse · posted 2026-07-20. ApplySarthi collects openings and links to application pages; the role is advertised by Lightning AI, not by us.