Liquid AI recovers 97% of accuracy in 4-bit models with distillation

Quantizing an LLM to 4 bits is the most direct way to run it on modest hardware: less memory, more throughput. The known problem is that this compression comes at a cost in quality. Liquid AI has just published, on the Hugging Face blog, Q4_0 checkpoints of the LFM2.5 models that, according to the company, recover 97% of the average accuracy lost in quantization without giving up the size or speed of the Q4_0 format.

The release covers four models: LFM2.5-230M, LFM2.5-350M, LFM2.5-1.2B-Instruct, and LFM2.5-2.6B. These are small models, designed to run at the edge, and that's where the technique makes a real difference.

The problem with post-training quantization

The most common way to reduce a model to 4 bits is post-training quantization (PTQ): the model, already trained in BF16, is taken and its weights are converted to a lower-precision representation. It's fast and doesn't require retraining, but it introduces rounding error in every weight. In large models this error gets diluted; in small models, where each parameter carries more responsibility, the degradation shows up in reasoning, instruction-following, and tool-use tasks.

That's exactly the gap Liquid AI targeted. The Q4_0 GGUFs that already existed for the LFM2.5 models were produced via PTQ. The new ones come from a different process.

How quantization-aware distillation works

The technique is called Quantization-Aware Distillation (QAD). It combines two ideas already known individually:

• Quantization-aware training: instead of quantizing afterward, the model is trained while already simulating 4-bit quantization during the forward pass. This way, the weights learn to live with the precision loss instead of suffering from it at the end.
• Distillation: a high-precision teacher model (the BF16 one) transfers its knowledge to a quantized student model. The student doesn't just learn the right answer, but the teacher's probability distribution, a much richer signal.

Combining the two makes the 4-bit student reproduce the behavior of the full-precision teacher. The result is a checkpoint that takes up the same space as a regular Q4_0, but thinks more like the original.

The numbers

Liquid AI compared the old GGUFs (PTQ) against the new ones (QAD) using BF16 as the reference ceiling within the format. The test suite covers reasoning, instruction-following, tool use, and agentic capabilities: GPQA Diamond, MMLU-Pro, IFEval, IFBench, Multi-IF, and BFCLv4. They also added a math evaluation scaled by model size (GSM8K for the smaller ones, AIME25 for the larger ones). Each number is the average of five runs.

Performance retention relative to BF16 came out at:

• 97.1% for LFM2.5-230M
• 96.5% for LFM2.5-350M
• 97.4% for LFM2.5-1.2B-Instruct
• 96.6% for LFM2.5-2.6B

More interesting than the average is the comparison with more generous quantization formats. The 230M and 350M QAD Q4_0 checkpoints match the quality of a Q5_K_M (which uses more bits) within the test's variance, delivering 4% to 33% more decode throughput. The 1.2B and 2.6B ones tie with Q4_K_M while running 3% to 14% faster. In short: the same quality as larger formats, but running faster and taking up less space. Liquid AI also says the checkpoints beat Unsloth's UD-Q4_K_XL, a strong external PTQ reference, where applicable.

Running it in practice

The files are standard GGUF and work with llama.cpp or any runtime that accepts Q4_0 artifacts. Downloading and running directly from Hugging Face is one line:

llama-cli -hf LiquidAI/LFM2.5-350M \
  --hf-file LFM2.5-350M-QAD-Q4_0.gguf \
  -p "What is C. elegans?"

All four models are already available on Hugging Face: LFM2.5-230M, LFM2.5-350M, LFM2.5-1.2B-Instruct, and LFM2.5-2.6B.

The point that matters for those building in Brazil is the target hardware. The company measured throughput on four platforms that are quite representative of what's available here: a MacBook Pro, the NucBox EVO-X2 mini PC, a Samsung Galaxy S26 Ultra, and a Raspberry Pi 5. The MacBook and NucBox use GPU; the phone and Raspberry Pi run on Arm CPU. In other words: it's possible to put a useful LLM to run on a Pi 5 or a phone without depending on the cloud, which solves latency, recurring API cost, and, above all, the privacy of data that can't leave the device.

Where it's not worth it

QAD is no silver bullet. The technique requires retraining with a high-precision teacher, which only makes sense for those who publish the models, not for the dev who wants to quantize just any model at home. For personal use, PTQ remains the practical path, and tools like Unsloth's are still relevant.

Another point: these are small models, from 230M to 2.6B parameters. They don't replace a GPT or Claude on complex tasks. The niche is clear: local agents, classification, extraction, embedded tools, and cases where running at the edge matters more than the raw capacity of a large model. In these scenarios, gaining speed without losing quality in a 4-bit format is exactly the kind of optimization that decides whether a project runs or not on the available hardware.

The technical takeaway is that the frontier of quantization is no longer just about squeezing bits after training, but about training while already accounting for the compression. If QAD scales to larger models while keeping this recovery rate, the cost calculation for running LLMs locally changes in a meaningful way.