Transformers now runs GGUF quantized models from llama.cpp natively

Hugging Face published a post on September 22, 2026, signed by Marc Sun, Arthur Zucker, and Lysandre Debut, announcing that the transformers library now loads and runs checkpoints in the GGUF format by reusing llama.cpp's own Metal kernels. In practice, this means taking a quantized model published by someone like Unsloth, LM Studio Community, or bartowski on the Hub and running it with AutoModelForCausalLM.from_pretrained, without converting anything and without leaving the API that anyone who already uses transformers knows by heart.

Why this matters if you already use Ollama or llama.cpp

GGUF is the format created by the llama.cpp team to pack weights, tokenizer, and chat template into a single file, with different quantization levels. It's the format behind Ollama, LM Studio, and Jan, and has already had millions of downloads on the Hub. Until now, if you wanted to use these same checkpoints inside a transformers pipeline (for fine-tuning, evaluation, debug hooks, or custom generation), the normal route was to convert the GGUF back to the safetensors format or maintain two separate tool ecosystems. The integration changes that: the GGUF file now becomes a valid direct input.

The gain isn't raw performance above plain llama.cpp (Hugging Face itself is clear: llama.cpp remains the recommendation when the goal is just efficient local inference). The gain is tooling: you can inspect intermediate activations with hooks, write a custom generation loop in Python, plug in your own LogitsProcessor, or take an already-quantized GGUF and keep training from it with GgufConfig(dequantize=True).

How to load a GGUF today

The initial requirement is specific: a Mac with Apple Silicon, one of the last two PyTorch versions, and the development version of transformers (support isn't in a stable release yet, only on the main branch), along with the kernels lib:

pip install -U "git+https://github.com/huggingface/transformers.git" kernels

Loading itself requires no extra configuration beyond pointing to the repository and the file:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "unsloth/Qwen3.5-4B-GGUF"
filename = "Qwen3.5-4B-Q4_K_M.gguf"

tokenizer = AutoTokenizer.from_pretrained(model_id, gguf_file=filename)
model = AutoModelForCausalLM.from_pretrained(model_id, gguf_file=filename)

From there on it's the standard API: apply_chat_template, generate, decode. When the weights stay packed and running on Metal, transformers automatically loads the compatible ggml kernels and uses ggml-org/ggml-attn as the attention implementation; if the kernel isn't available, it falls back to sdpa with a warning (and you can force this manually). Without a compatible quantization kernel, the loader unpacks the entire model before running, which consumes significantly more memory, a detail that changes the calculation of how much unified RAM you actually need.

You can also spin up an OpenAI-compatible server directly from the quantized checkpoint:

transformers serve "unsloth/Qwen3.5-4B-GGUF:Qwen3.5-4B-Q4_K_M.gguf"

The :.gguf syntax solves the problem of repositories that host several quantizations of the same model. The resulting endpoint at http://localhost:8000/v1 talks to any client that speaks the OpenAI API, including Jan or Pi, by pointing the Model ID to the same repository:file pair.

How much each quantization level costs

The post brings file size numbers for Unsloth's Qwen3.5-4B, which help decide where to start:

| Variant | Size | Trade-off | |---|---|---| | BF16 | 8.42 GB | baseline without quantization | | Q6_K | 3.53 GB | more precision than the smaller ones | | Q5_K_M | 3.14 GB | size/precision middle ground | | Q4_K_M | 2.74 GB | practical starting point |

Hugging Face's own recommendation is to start at Q4_K_M and move up to Q5_K_M or Q6_K if there's memory to spare, always evaluating on the actual task the model will perform, because the quality loss from aggressive quantization varies from model to model.

The benchmark against llama.cpp

The performance comparison was done on a MacBook Pro M2 Max with 32 GB of unified memory, macOS 26.6, PyTorch 2.12.1, and kernels 0.17.0, plugged into the wall. On the llama.cpp side, the number comes from llama-bench (build 5f55650a7, release b10200, ggml 0.18.0 Metal backend) running llama-bench -m -p 0 -n 128 -r 3, measuring only the token generation rate (tg128), without prompt processing. On the transformers side, the measurement is a full generate producing the same 128 tokens from a 12-token prompt, best of three warmed-up runs, including prefill. The team itself warns that the conditions aren't identical (one includes prefill, the other doesn't), but the result presented is that transformers comes close to llama.cpp on the three checkpoints tested: a small dense model, a larger dense model, and a mixture-of-experts.

This result isn't framework magic: it's direct reuse of ggml's Metal kernels. The kernels lib distributes builds of these kernels through the Hub, and transformers now calls them. There are five pieces: ggml-quantization (reads quantized weights without expanding the entire matrix before each decode), ggml-norm (fuses normalizations, including Qwen3.5/Qwen3.8's zero-centered RMSNorm), ggml-attn (flash attention on Metal), ggml-gated-delta-net (speeds up Qwen3.5/3.8's hybrid linear attention layer), and a Hugging Face-built topk for expert routing in MoE models.

Two optimizations that apply to any model

Besides the kernels, the post describes two changes to the generate loop that aren't specific to GGUF and now apply to any transformers model: removing the attention mask early when there's no padding (PR #48814) and delaying the stopping-criteria check so the CPU keeps scheduling work while the GPU decodes (PR #47975). These are CPU/GPU synchronization tweaks, the kind of thing that doesn't show up in a flashy changelog but reduces the dead time between generated tokens, especially relevant for anyone who was already complaining about high latency running larger models locally.

Where this doesn't reach yet

The list of stated limitations is straightforward: the packed inference path only works on MPS for now, so anyone relying on CUDA on Linux or Windows gets none of this today, only the dequantization route, which consumes more memory. Batching with padding still doesn't have the same performance gain as inputs without padding, and architecture coverage is restricted to Qwen3.5 (dense and MoE) and compatible Qwen3.8 checkpoints. In other words: if your use case is Llama, Mistral, Gemma, or any other family outside this list, or if you need batch throughput on a production CUDA GPU, this integration doesn't solve it yet. Hugging Face asks for issues in the repository with the checkpoint and use case to prioritize expanding coverage.

Who this is for in practice

The scope is clear: this doesn't replace llama.cpp or Ollama for anyone who just wants to run a fast local model, and Hugging Face itself recognizes llama.cpp as the efficiency benchmark. The value shows up in hybrid scenarios, like evaluating the quality of a GGUF with the evaluation workflows you already maintain in transformers, validating whether a GGUF conversion preserved the weights correctly by comparing it with the original checkpoint, prototyping a custom logits processor without rewriting the generation logic in another language, or taking a quantized checkpoint and continuing fine-tuning from it. For Brazilian teams running prototypes on MacBooks before deciding whether it's worth spinning up a cloud GPU, the integration reduces the friction of maintaining two tool pipelines (one for experimenting in transformers, another for serving locally with GGUF), but the MPS-only support and coverage limited to Qwen3.5/3.8 make clear that this is still an early piece, not a production stack replacement.