Same Cluster, 33 Points More Utilization: What Changed Was the Order

GPU spend has become a line item in the budget of any team running AI in production, including in Brazil, where accelerator instances are expensive and often scarce in the local regions of the cloud providers. A study published on the Hugging Face blog by the Dharma AI team, titled Same Cluster, 33 Points More Utilization: What Changed Was the Order, argues something uncomfortable: most of the waste isn't in the hardware or the model, it's in the order in which allocation decisions are made.

The thesis is straightforward. The authors built a constraint-aware allocator and compared it against a FIFO scheduler across seven benchmark scenarios, on the same hardware and with the same workloads. GPU utilization rose by up to 33 percentage points, and priority-weighted output (output weighted by priority) rose in every scenario, reaching 105%. Nothing changed about the machine. What changed was only who enters the grid first.

The Decision FIFO Doesn't Know How to Make

The text starts by dismantling the cliché of "keeping the GPUs busy." That isn't an executable decision. The real decision is narrower and harder: which GPU runs which job, at which timestep, with which priority. Formally, it's a binary choice per GPU-job-timestep combination, and the result is a grid with a job's name in each cell, or nothing.

Four types of workload compete for this grid: training, real-time inference, batch inference, and quantization. They split into two incompatible allocation shapes. Training, batch, and quantization are "batch-like": once started, they need a contiguous block of GPUs held without interruption until they finish. Real-time inference, on the other hand, is elastic, driven by a demand curve that changes at every timestep. Two incompatible shapes fighting over the same hardware at the same instant is the core of the problem.

Where FIFO Bleeds Under Contention

The baseline uses a reasonable policy when there's slack: real-time inference served from a fixed reservation, and everything else placed on a first-come, first-served basis. The problem appears under contention, and it costs in two ways.

The first is the reservation. Since a first-come, first-served scheduler has no mechanism to release GPUs during a traffic valley and reclaim them before the next peak, the only way to guarantee availability is to reserve the entire day's maximum peak. An application that needs six GPUs at noon and two at 4 a.m. holds all six for 24 hours. The four idle ones remain unavailable to any batch job. That's why the baseline sits close to half the cluster in reservation-dominated scenarios (51.6% in the mixed control, 53.6% in the training-heavy case).

The second is the order itself. Under real contention, which jobs fit depends on the placement sequence, not just how much capacity exists. FIFO places each job on arrival, without weighing how much it's worth or checking what still needs to fit within the horizon. The result: high-priority work waits behind whoever asked first, and capacity gets committed to positions that later jobs can't use. The authors' analogy is a good one: it's like an airline handing out aircraft to whoever called first, only to find there's no plane left for the route that actually pays.

What the Allocator Does Differently

The fix has two moves. Real-time demand starts being treated as a curve, not a ceiling: allocation is made against each timestep's demand, with batch work filling the valleys, capped by a limit on how many GPUs a real-time job can swap between consecutive timesteps. And batch-like jobs are placed by priority across the entire horizon, not by order of arrival.

The formal model defines five constraints for a legal allocation: at most one job per GPU per timestep; respecting the demand band, with whatever's already running being inherited and kept; contiguous blocks sized in powers of two; a swap cap for real time; and nothing already started can be interrupted. The objective function has two terms: allocating a GPU to a batch job yields a reward equal to its priority times a time-decay weight; failing to meet real-time demand generates a penalty proportional to the shortfall.

The detail that ties it all together: the real-time penalty weight is 5 to 10 times larger than the allocation weight. One unit of unmet real-time demand costs what 5 to 10 GPU-timesteps of equivalent-priority batch work cost. It's this asymmetry that makes it safe to lend GPU to batch during the valley, because underserving real time afterward is so expensive that the penalty, rather than a static reservation, protects availability. The time weight decays because, by the next scheduling round, new jobs will have arrived: capacity used now is worth more than capacity promised later.

Heuristic on the Hot Path, Formal Model Behind It

Combinatorial allocation of this kind is NP-hard, and the scheduler gets re-invoked on every job arrival, so the answer has to come back within the interval between two API requests. That's why the architecture puts a heuristic on the hot path and the formal model behind it, as a specification. The heuristic isn't a generic greedy algorithm: its rules are the model's structural constraints, so every grid it produces is legal by construction. It runs in 1 to 2 ms in contended scenarios and 15 ms with 64 GPUs and 30 jobs. There are two modes: fast returns the heuristic's grid; full uses that grid as a starting point for the formal model to try to improve on it, meant for periodic review.

The Numbers, and the Scenario That Separates Occupancy From Value

The seven published results:

• Mixed control (8 GPUs, 10 jobs): 51.6% → 72.4%, value +54.8%, 1 ms
• Real-time contention (8 GPUs, 8 jobs): 75.0% → 80.2%, +24.6%, 1 ms
• Training-heavy (8 GPUs, 16 jobs): 53.6% → 87.0%, +105.1%, 2 ms
• Large mixed (14 GPUs, 16 jobs): 76.8% → 82.7%, +43.8%, 2 ms
• Oversubscribed (8 GPUs, 9 jobs): 85.4% → 87.5%, +33.6%, 1 ms
• Scale test (64 GPUs, 30 jobs): 44.9% → 44.9%, +15.9%, 15 ms
• Uniform priority (14 GPUs, 16 jobs): 76.8% → 87.5%, +23.1%, 2 ms

Two scenarios deserve attention. In the scale test, FIFO and the allocator produce identical utilization (44.9%) and the same throughput (27 of 30 jobs completed), but the allocator delivers 15.9% more priority-weighted value. It's the central argument in measured form: occupancy doesn't say what the cluster is gaining. Two identical dashboards can hide materially different outputs.

The uniform priority test answers the obvious skepticism. If you force every job to the same priority, the allocator still moves utilization from 76.8% to 87.5% and value by 23.1%. In other words: the gain isn't just an artifact of ordering by priority, planning placements across the horizon contributes on its own.

Demand Forecasting and the Sliding Horizon

None of this works if the demand numbers are wrong, and they are forecasts, not inputs. Dharma AI uses estimators specialized by workload type. The training forecaster conditions on 22 features, including a categorical variable with 10 training variants (LoRA cuts trainable parameters by up to 10,000x and memory by ~3x versus full fine-tuning; DPO eliminates the reward model and the RLHF sampling loop). Quantization gets its own forecast by calibration tier and algorithm (bitsandbytes, AWQ, GPTQ). Real time becomes a weekly demand profile, continuously recalibrated, under the same swap cost the optimizer applies, so that the forecast and the optimizer don't fight each other.

Against forecast error, the architecture uses a sliding horizon: it optimizes 24 hours ahead but commits only the current timestep and reruns every 30 to 60 minutes. The plan for the following hours exists only so that the current decision knows there's a future, avoiding the end-of-world effect. Error gets absorbed by reoptimization instead of compounding.

What Remains Open

The numbers come from a synthetic benchmark built for genuine contention, with a single-ordering baseline, not a production A/B study with a real customer's live traffic. For Brazilian teams, the practical lesson doesn't depend on adopting Dharma AI's code: if your cluster reserves GPU for the day's peak and schedules by arrival, there's likely idle capacity that a priority- and block-shape-aware policy would recover, without buying more hardware. Where the effort isn't worth it is exactly where FIFO is already acceptable: a cluster with slack, where everything fits regardless of order. The obvious recommendation is to measure your contention rate before swapping schedulers. The authors provide open-source demos and models on Hugging Face for anyone who wants to reproduce the scenarios.