Impactful scheduling for GPU clusters

Ai2 replaced its priority-based scheduler with a system using GPU time budgets, hierarchical fair-share allocation, and a time-slicing contract. The institute manages thousands of NVIDIA H100, B200, and B300 GPUs across clusters of 88 to 1024 GPUs serving roughly 150 researchers, with demand at 2-3x available capacity. The priority approach produced GPU "squatting" and priority inflation, with all workloads eventually marked HIGH priority. The new model shifts allocation decisions to administrative budgeting, letting leadership fund research efforts with GPU time.