Ai2 fixed GPU squatting by turning priority into a budget

Ai2's infrastructure team runs GPU clusters for around 150 researchers, where demand runs two to three times supply. Their write-up is honest about what the old priority-based scheduler did to them: people squatted on GPUs they were not using, priority kept inflating until lower tiers starved, and someone had to negotiate by hand to shut down jobs that could not be preempted. The fix was to stop treating access as a ladder of priorities and start treating GPU time as a budget that managers allocate to programs, projects, and individuals. Any protected, non-preemptible work has to be charged against that budget, so trying to game the scheduler now costs you your own allocation.

Two more pieces make it work. A hierarchical fair-share algorithm tracks usage over a rolling seven-day window and favors allocations that have been under-served, with the tree shaped around Ai2's actual program structure. And every job declares a minimum protected runtime, capped at eight hours, after which it can be preempted and requeued. Before rollout they tested the design in a simulator against historical and hand-built scenarios.

The reported results, from a 30-day window on their own clusters, are specific. Teams received 98% of the GPU hours they were owed, and 13 of 15 allocations got at least 95%. The p90 wait for a debug job fell from about two hours to 30 seconds. On the largest H100 cluster the median wait dropped from five minutes to 24 seconds. Repairs that needed a human to step in fell 74%. The eight-hour cap did hurt interactive sessions, so they are planning a CPU-only dev cluster and restorable sessions.

Why it matters

If you run shared GPUs, this is a concrete alternative to the priority wars that waste both hardware and goodwill. Budgets make hoarding expensive to the hoarder, and the numbers here give you a baseline to argue for the change, with the caveat that they come from one org's short test.