Slurm ☆

Slurm has scheduled supercomputers for two decades — gang scheduling, topology-aware placement, fair-share queues — which is exactly what a 10,000-GPU training job needs. Most frontier labs run Slurm (sometimes atop Kubernetes) for training while serving lives cloud-native.