AIARTICLE

Fine-tuning takes longer than the FLOPs math promises, study of 272 real jobs shows

Fine-tuning takes longer than the FLOPs math promises, study of 272 real jobs shows
Image: Alan Andrade
Article · AI
Reading
6 min
Tables
2
In short

A Dharma-AI study of 272 real fine-tuning jobs on H200 GPUs shows that the classic FLOPs formula is off by hours, and proposes a model that gets it right within minutes.

The math every fine-tuning team has already done

Anyone who fine-tunes custom models knows the formula: multiply the number of parameters (P) by the number of training tokens (D), multiply by 6, divide by GPU throughput adjusted by a utilization factor called MFU (Model FLOPs Utilization). The result comes out in hours and becomes the estimate that goes into the project schedule.

The problem, according to a Dharma-AI study published on November 18, 2026 on the Hugging Face blog, is that this math is badly off. Running the formula against 272 real supervised fine-tuning (SFT) jobs, the researchers found an R² of just 0.31 and a mean absolute error of 12.41 hours, even using the best MFU value tested (30%). For jobs that last 23 hours on average, that's an error the size of the job itself.

272 real jobs, 7,382 hours of H200 GPU time

The Dharma-AI team (the post is signed by Bruno Duarte, Gabriel Pimenta de Freitas Cardoso, Isis F Carvalho, Breno de Almeida Beleza, and Felipe Lima) used operational data from a production cluster: between 8 and 32 NVIDIA H200 GPUs spread across 1 to 4 nodes, running alongside other training and inference workloads. That means real network and storage contention, not an isolated lab environment.

The 272 runs cover model families with up to 15 billion parameters, both LoRA and full fine-tuning, with Hugging Face Transformers and Unsloth stacks treated as a categorical variable, without assuming a fixed speed gain between the two. Durations ranged from 1.6 to 172.4 hours, with a mean of 23.43 hours and a median of 15.29 hours.

What the FLOPs formula doesn't see

The equation that divides 6·P·D by the product of the number of GPUs, peak throughput, and MFU describes the optimization loop well, but a fine-tuning job is much more than the loop. Before the first training step, there's environment setup, model download, and tokenization. During training, there are periodic evaluations and checkpoint uploads. At the end, the final model goes to object storage, competing for I/O with other jobs on the cluster.

None of this scales predictably with compute volume. Two jobs with the same theoretical FLOPs cost can finish hours apart, depending on how many times they evaluate the model, how many checkpoints they save, and how much concurrent network traffic they face.

To avoid stacking the comparison in its favor, the FLOPs baseline used in the study already counts scheduled evaluation tokens in addition to training tokens, and varies MFU from 0.25 to 0.65 to give the formula the best possible chance.

From R² 0.31 to R² 0.98

The proposed alternative treats duration as a supervised regression problem: given a feature vector extracted from the job's configuration (hyperparameters, dataset size, model identifiers, GPU allocation), predict how much time will pass between the GPU being assigned and the job finishing. All of this is known before the job starts running, which is what makes the prediction useful for whoever schedules the cluster queue.

The researchers tested Linear Regression, Ridge, XGBoost, and CatBoost, and also compared them against Calculon (Isaev et al., SC '23), an analytical LLM training performance model that the authors extended to cover LoRA, grouped-query attention, fused attention, bfloat16, and SwiGLU. The results under 5-fold cross-validation:

ModelR²RMSE (h)MAE (h)
FLOPs (MFU 30%)0.3120.4512.41
FLOPs (MFU 45%)0.1822.3813.01
FLOPs (MFU 60%)0.0124.5614.62
Calculon-0.5130.4020.36
Linear Regression0.6913.527.52
Ridge0.7312.727.23
XGBoost0.974.181.95
CatBoost0.983.521.71

Calculon, even though it's a much more detailed analytical model than the FLOPs math, with an explicit layer graph, memory traffic, and inter-device communication, came out worse: its estimates correspond to a median of just 11% of the observed real time. That doesn't invalidate Calculon as a training-step analysis tool, which is what it was designed for. But it shows how much of a production job happens outside the optimization step itself.

The final CatBoost uses 24 features, selected from 18 candidate sets through correlation filtering, mutual information, and permutation importance, followed by a grid search with MAE as the criterion.

The signals that matter most

CatBoost's feature importance is concentrated in a handful of variables, and not all of them are the obvious ones. num_eval_data (total samples processed in evaluation over the job's lifetime) leads, followed by num_trainable_params, batch_eval_size_percent, num_all_params, and the interaction num_gpus*batch_train_percent, with num_params_saved close behind.

In short: evaluation frequency, evaluation set size, and how many checkpoints the job persists weigh as much as model and dataset scale. These are configuration choices ML teams often make out of habit, without measuring their impact on total time.

One point worth noting in the SHAP charts: larger GPU allocations show up with a positive contribution to predicted time. That doesn't mean adding GPUs makes the job slower. In the dataset, larger jobs tend to use more GPUs, so num_gpus works partly as a proxy for job scale.

Feature importance chart shows num_eval_data, num_trainable_params, and num_gpus among the variables with the highest average contribution to the model's predicted time
Feature importance chart shows num_eval_data, num_trainable_params, and num_gpus among the variables with the highest average contribution to the model's predicted time. Reprodução: huggingface.co.

Where the prediction still gets it wrong

Looking at the 272 out-of-sample predictions, the median absolute error is 0.51 hours; 80% of predictions fall within 2.20 hours of the real value, and 90% within 4.31 hours. But the error isn't uniform across duration ranges:

Job duration (h)RunsMedian error (h)P90 (h)
1.56 to 5600.301.55
5 to 1063-1.27
10 to 30690.734.17
30 to 60671.065.50
60 to 175135.1918.51

Up to 60 hours of duration, the median error stays around one hour or less. Above that, the error margin grows considerably, but there are only 13 runs in that range in the dataset, which the authors themselves treat as a data limitation rather than a method limitation. In a separate test, increasing the training data fraction from 85% to 100% already reduced MAE from 1.63 to 1.50 hours, with no sign of a plateau, suggesting that more historical data would still help.

What's left out, and what changes for those training models here

The study has a narrow scope by design: it covers only supervised fine-tuning, no RLHF, DPO, or RLVR, only H200 GPUs (the calibrated model shouldn't be expected to work numerically on another generation without retraining), and it has few very long runs. The authors themselves already point to next steps: hardware descriptors for mixed pools, transfer learning for configurations with few observations, and more data from long runs.

For those who build or operate GPU clusters in Brazil, the practical takeaway doesn't depend on reproducing the entire paper. The lesson is that the FLOPs math will never capture the cost of evaluation, checkpointing, and shared I/O, so any deadline estimate based on it alone carries a systematic downward bias. Teams that log evaluation configuration, checkpoint policy, and per-GPU load from their own jobs already have the raw material to train a similar regressor, instead of inheriting the optimistic estimate from a textbook formula.

The full paper, "Predicting End-to-End Fine-Tuning Time for Generative Language Models," should be out soon on arXiv according to Dharma-AI, with the feature formulas, the selection process, and the final hyperparameters detailed for anyone who wants to rebuild the pipeline on top of their own execution logs.

Translated from the Brazilian Portuguese original · Read the original

View profile →