GGUF in Hugging Face Transformers: when it's worth switching from llama.cpp

GGUF is no longer llama.cpp's exclusive territory

GGUF is the single file format used by the GGML ecosystem (the foundation of llama.cpp) to store metadata and tensors for an already-quantized model. Until recently, if you wanted to run a .gguf file, the natural path was to compile or install llama.cpp and point the binary at the file. The current Transformers documentation (on the main branch, which requires installing from source, unlike the latest stable pip release, 5.17.0) shows that you can now load these same files directly with AutoModelForCausalLM, pulling the repository from the Hub as if it were any other checkpoint.

This is good news for anyone already living inside the Transformers ecosystem who doesn't want to maintain two runtimes (one in Python for fine-tuning/evaluation, another in C++ for serving). But before deciding to migrate a production service that currently runs on llama.cpp, there's fine print that changes the math: the memory savings from the quantized format aren't automatic for every architecture.

Two paths inside Transformers: packed and dequantize

The Transformers loader can follow two paths when you pass a GGUF file:

• Packed: the weights stay compressed in memory, the way they were in the file, and matrix multiplications run directly on the quantized blocks. This is what actually reproduces the RAM savings llama.cpp has always offered.
• Dequantize: the model is fully decompressed at load time, leaving you with a normal dense model, the same size it would be with no quantization at all.

The detail that changes everything: the packed path today is only available for Qwen3.5 and Qwen3.5 MoE, and only when the ggml-org/ggml-quantization kernel is accessible via the kernels package. Every architecture outside that list (Llama, Mistral, Qwen2, Qwen2Moe, Phi3, Bloom, Falcon, StableLM, GPT2, Starcoder2, among others) goes through the legacy loader, which always dequantizes. In other words: you download the small file from the Hub, but the model loaded on the GPU or CPU takes up the memory of a full-precision model.

There's another hardware-specific detail: the documentation states that the loader defaults to MPS (Metal, i.e., Apple Silicon) when the packed kernel is present, and that the packed path automatically uses float32 "because it's faster on MPS." That's a strong sign this path was designed with Mac in mind first. If your production service runs on Linux with CUDA, it's worth testing explicitly before assuming you'll get the same memory savings you'd get on a MacBook.

From the Hub to the loaded model

The first step is installing the kernels package; without it, the packed path isn't even attempted and the loader falls straight back to full dequantization:

pip install kernels

After that, loading works just like any other Transformers model, the only difference is the gguf_file parameter:

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "unsloth/Qwen3.5-4B-GGUF"
filename = "Qwen3.5-4B-Q4_K_M.gguf"

model = AutoModelForCausalLM.from_pretrained(model_id, gguf_file=filename)
tokenizer = AutoTokenizer.from_pretrained(model_id, gguf_file=filename)

The same pattern works for Qwen3.5 in mixture-of-experts format:

moe_model_id = "unsloth/Qwen3.5-35B-A3B-GGUF"
moe_filename = "Qwen3.5-35B-A3B-Q4_K_M.gguf"

moe_model = AutoModelForCausalLM.from_pretrained(moe_model_id, gguf_file=moe_filename)
moe_tokenizer = AutoTokenizer.from_pretrained(moe_model_id, gguf_file=moe_filename)

If you want to force dequantization even on a model that would otherwise have the packed path available, for example to run in bf16 on a CUDA GPU where the packed kernel doesn't apply, the documentation itself exposes this via GgufConfig:

import torch
from transformers import AutoModelForCausalLM, GgufConfig

quantization_config = GgufConfig(dequantize=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id, gguf_file=filename, quantization_config=quantization_config, dtype=torch.bfloat16
)

For attention, when the packed kernel is available on MPS, the default switches to ggml-attn, the same flash-attention kernel llama.cpp uses for prefill and decode. Explicitly passing attn_implementation always takes precedence over that default: if you want to keep sdpa (PyTorch's default) for compatibility with the rest of your pipeline, just declare attn_implementation="sdpa". The snippet below does the opposite, explicitly forcing ggml-attn regardless of what the loader would choose by default:

model = AutoModelForCausalLM.from_pretrained(
    model_id, gguf_file=filename, attn_implementation="ggml-org/ggml-attn"
)

Packaging it into a FastAPI endpoint

From here to production, the path is the same as serving any other Transformers model: load it once at process start and reuse the instance on every request. A reasonable skeleton:

from contextlib import asynccontextmanager
from fastapi import FastAPI
from pydantic import BaseModel
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL_ID = "unsloth/Qwen3.5-4B-GGUF"
FILENAME = "Qwen3.5-4B-Q4_K_M.gguf"

state = {}

@asynccontextmanager
async def lifespan(app: FastAPI):
    state["tokenizer"] = AutoTokenizer.from_pretrained(MODEL_ID, gguf_file=FILENAME)
    state["model"] = AutoModelForCausalLM.from_pretrained(MODEL_ID, gguf_file=FILENAME)
    yield
    state.clear()

app = FastAPI(lifespan=lifespan)

class Prompt(BaseModel):
    text: str
    max_new_tokens: int = 256

@app.post("/generate")
def generate(payload: Prompt):
    tokenizer = state["tokenizer"]
    model = state["model"]
    inputs = tokenizer(payload.text, return_tensors="pt")
    output = model.generate(**inputs, max_new_tokens=payload.max_new_tokens)
    return {"text": tokenizer.decode(output[0], skip_special_tokens=True)}

This skeleton gets you to "it works," but it's still much simpler than a dedicated inference server typically is in production: there's no request queue, no native token-by-token streaming, and no fine-grained per-session context control. If your traffic is serious, the natural next step here is to run this process behind a single-worker uvicorn (the model can't be duplicated per worker without duplicating memory) and handle concurrency with an async queue in front, not multiple processes.

The shortest route: transformers serve

If you only need a compatible endpoint for testing or an internal service, Transformers already bundles a serving command that spares you from writing the FastAPI by hand. Each .gguf file in a repository is exposed as its own model, addressed as <repo>:<file>.gguf:

transformers serve unsloth/Qwen3.5-4B-GGUF:Qwen3.5-4B-Q4_K_M.gguf

It's the right path for quick prototyping. For production, the dedicated FastAPI still wins on a point that matters: authentication, observability (per-route latency metrics, structured logs), and rate-limit policies are things you need to bolt on regardless, and it's simpler to do that on top of an application you already control than to try to fit it into the generic serving command.

What decides the switch

The question that matters before abandoning llama.cpp in a service that's already in production isn't "does Transformers load GGUF?" (it does), it's "do my model and my hardware fall into the packed path?" If the answer is no, you're trading a runtime designed to run quantized from start to finish for a runtime that downloads the small file and, in practice, gives you back the memory footprint of a full-precision model. That can be an acceptable price if the gain is unifying your inference stack inside the Python ecosystem you already maintain for fine-tuning and evaluation, but you need to measure the process's resident memory before assuming the savings, especially outside Qwen3.5 and outside Apple Silicon. It's also worth confirming which installed version of transformers already has these parameters (GgufConfig, attn_implementation="ggml-org/ggml-attn"), since the documentation cited here is from the development branch, not the latest stable version published on PyPI.