Senior AI Infrastructure Engineer - HPC & Compute Clusters

Job details

Work model

Hybrid

Salary

Not listed by source

Location

Toronto

Job description

ABOUT US Veeda AI is building the next generation of multimodal foundation world models for Physical AI. We're a small, fast-moving team of engineers and researchers from leading AI labs, tackling some of the most challenging problems at the intersection of AI, robotics, and embodied intelligence. If you're excited about pushing the boundaries of what's possible with Physical AI, you'll have the opportunity to make an outsized impact from day one. RESPONSIBILITIES - GPU Cluster Orchestration: Design, deploy, and maintain large-scale bare-metal GPU clusters running Slurm and Kubernetes to power multi-node distributed training runs. - Workload Management & Scheduler Tuning: Configure, tune, and optimize Slurm schedulers, dynamic job queuing, priority topologies, and autoscaling for optimal GPU utilization and job throughput. - Kubernetes Integration: Manage K8s clusters and hybrid Slurm-K8s environments (e.g., KubeRay, MPI Operator, Slurm on K8s) for serving, data processing pipelines, and interactive research environments. - Storage & Network Performance: Optimize high-speed interconnects (InfiniBand/RoCE, NCCL, NVLink) and high-throughput distributed storage (Lustre, WEKA, Ceph) to keep tens of thousands of GPUs continuously saturated. - Reliability & Monitoring: Build observability pipelines (Prometheus, Grafana, DCGM) to proactively detect hardware degradation, GPU silent errors, network flaps, and node failures before they impact training jobs. REQUIREMENTS - You have a Bachelor's degree in Computer Science, Computer Engineering, or equivalent hands-on experience in high-performance computing (HPC) or infrastructure engineering. - You have deep hands-on expertise administering Linux-based HPC clusters running Slurm or Kubernetes (job scheduling, cgroups, fair-share policies, topology configuration). - You possess strong troubleshooting skills in low-level Linux networking, kernel parameters, hardware diagnostics, and storage systems (e.g., NFS, NVMe-oF, or distributed filesystems). - You are proficient in automation and infrastructure-as-code tools (e.g., Ansible, Terraform, Helm, Python/Bash scripting). - Background supporting distributed deep learning frameworks (DeepSpeed, Ray). NICE TO HAVE - Experience managing high-density GPU infrastructure (NVIDIA H100/B200/B300 systems, DGX/HGX architectures, DCGM, NCCL tuning). - Experience with high-speed network fabrics including InfiniBand (Subnet Manager, SMDB) and RoCE (v2).

• Staff CAD Platform Engineer (NX / Teamcenter)Anduril Industries · Remote
• Senior Supplier Industrialization Engineer, ElectricalAnduril Industries · On-site
• Electrical Engineer, Air Dominance and StrikeAnduril Industries · Not provided by source
• Robotics EngineerAnduril Industries · On-site
• Cloud Deployment Engineer, SpaceAnduril Industries · On-site
• Staff Mechanical Engineer, SpaceAnduril Industries · On-site