AIARTICLE

Transformers runs 180-billion-parameter MoE model on a Mac with 32GB of RAM

Transformers runs 180-billion-parameter MoE model on a Mac with 32GB of RAM
Image: Alan Andrade
Article · AI
Reading
6 min
Tables
3
Code
2
In short

A project published by Hugging Face teaches the Transformers library to fetch from disk only the pieces of an MoE model that each token actually needs, reaching 14 tokens per second on a Mac with 32GB of memory.

The obstacle: a 180-billion-parameter model doesn't fit anywhere

Qwen3.8-Flash-Next is an MoE (mixture of experts) model with 48 layers of 512 experts each, of which only 10 are activated per token. In FP8, the weights add up to 121 GB, plus a 51 GB n-gram embedding table. No consumer laptop has enough unified memory for all of that at once.

This is the problem Tarek Ziadé, a researcher at Hugging Face, describes in a post published on the company's official blog on November 18, 2026. The idea: get Transformers, the most widely used Python library for running models, to load only the experts the router actually picks for each token, fetching the rest from disk on demand.

Why MoE offload isn't like any other offload

In MoE models, experts account for more than 90% of total weights, but each token uses only a small fraction of them. That should make it possible to run huge models on modest GPUs, as long as only the right experts enter memory at each step. In practice, doing this without killing speed is the hard part.

Before building his own solution, Ziadé mapped out how three engines already handle this offload:

EngineWhere experts liveWhat crosses the bus per tokenLimitation
llama.cppmapped via GGUF mmap, pinnable to CPU (--cpu-moe, --n-cpu-moe N, -ot "exps=CPU")only activations, never weightscache thrashing when experts don't fit in RAM
vLLMpinned RAM (--cpu-offload-gb N)the hot expert's weight, again on every token, via PCIeno cache and no disk layer
Transformers + Accelerateoffload folder, one whole module at a timethe layer's entire expert bank, even though only 8 to 10 are usedno notion of routing

The benchmark used Qwen3.6-35B-A3B on a 24 GB A10G GPU, with plenty of RAM to spare. The result explains why the standard Transformers path was unviable for this type of model:

EngineDecode (tok/s)
Strata, Q8_082 to 88
llama.cpp, Q8_0, --fit on63 to 64
vLLM, FP8, --cpu-offload-gb 16 --cpu-offload-params experts20.1
vLLM, BF16, --cpu-offload-gb 503.3
Transformers + Accelerate0.16

The difference between 0.16 and 82 tokens per second isn't about which engine is "better": it's about who knows how to reuse what's already in memory from one token to the next, and who recopies everything every time it's needed.

The core idea: a slot pool the original kernel already understands

The first version of Ziadé's project kept experts on disk with an LRU cache and ran each one in a Python loop, expert by expert. It worked, but landed at 10.1 tokens per second against 32 for vanilla Transformers running the whole model in memory: the bottleneck wasn't the disk, it was the loop, because the standard code uses grouped_mm, a kernel that processes all the experts in a layer at once.

The solution was to replace the cache with a slot pool: a single tensor, pre-allocated on the GPU, shared across all layers, plus a per-layer table that maps each expert's id to a physical slot. By remapping the router's ids to slots, the pool ends up looking like a normal expert bank, and the original grouped_mm runs on it without needing a custom matmul. This change alone took OLMoE-1B-7B, used as the small test model, from 10.1 to 34.6 tokens per second, against 38.1 for vanilla.

Putting it into production: experts_offload

This loader became an option on from_pretrained, proposed in pull request huggingface/transformers#50717. To test it, you need to install Transformers directly from the PR's branch:

bash
pip install "git+https://github.com/huggingface/transformers.git@refs/pull/50717/head"

With that, loading Qwen3.8-Flash-Next looks like this:

python
path = snapshot_download("tarekziade/Qwen3.8-Flash-Next-expert-streaming")

model = AutoModelForCausalLM.from_pretrained(
 f"{path}/dense",
 dtype=torch.bfloat16,
 device_map={"": "mps", "experts.*": "disk"},
 quantization_config=MetalConfig(bits=8),
 experts_offload={
 "cache_bytes": 12 << 30,
 "quantized_experts": f"{path}/experts-q3",
 },
)

Experts mapped to "disk" skip Accelerate and go through the streaming path: eight I/O threads read one contiguous block per expert, and the routers of the next two layers run ahead on the current state to guess which experts to prefetch. This combination of 3-bit quantization for the experts, 8-bit dense weights, and predictive reading is what makes the final result practical.

The result on the 32GB Mac, and the price of getting close to Strata

After conversion (which takes about five hours), the model uses roughly 18 GiB at rest and peaks at 25.4 GiB with a 6,000-token prompt, the practical limit before macOS starts swapping memory to disk. Compared to Strata, an MoE-specialized engine created by Niko Veit and cited as a reference in the post, Transformers' final numbers look like this:

MetricTransformersStrata
Decode, 200 tokens13.9 tok/s14.8 tok/s
Prefill, 4K prompt228 tok/s220 tok/s
Decode after 6K document12.4 tok/s13.7 tok/s

That's about 6% behind Strata on short decode, with slightly faster prefill. To get there, Ziadé reports specific tweaks: forcing GPU work to commit every block (because PyTorch's MPS backend queues up small kernels until the next sync), writing his own Metal kernels for parts with no fast path on macOS, adopting Strata's own GGUF weights to cut per-token data traffic, and using larger prefill chunks.

The bottleneck that's left, according to the author, is the back-and-forth between host and GPU: about 10ms per token, most of it with the GPU idle while waiting for Python to queue up the next layer. Fixing that for good would require control over per-layer synchronization, something PyTorch doesn't yet expose on the MPS stream.

What this changes for developers

In short: this isn't a finished product, it's a PR under discussion that proves a concept. The experts_offload API is still being negotiated with the Transformers maintainers, and the way experts are quantized is expected to migrate to the per-layer quantization plan from another PR (#50245). Anyone wanting to reproduce it needs to install directly from the branch, convert the model (five hours), and accept that it's still slower than dedicated alternatives like Strata or llama.cpp when the experts fit in RAM.

The author himself is direct about the practical limit: 14 tokens per second is too fast to brag about, but still too slow for the way he uses coding agents, which keeps running on a tower with two RTX 4090s at home. The project's value lies elsewhere: showing that it's possible to run a 180-billion-parameter model, of which 121 GB are experts, on an ordinary Mac, using only Python, Metal kernels compiled at runtime with torch.mps.compile_shader, and no C++ extensions.

For those evaluating running large models outside the cloud in Brazil, where datacenter GPUs are priced in dollars and the internet for remote inference has variable latency, this expert-streaming technique is what separates "I need an 80 GB GPU" from "a 32 GB laptop can handle it, just slower." It's worth testing in prototyping projects or for learning about systems, not yet in production where every second of agent latency matters.

It's also worth noting a cross-cutting limitation: vllm-metal, which brings expert offloading to Apple GPUs via MLX and follows similar logic, can't run this same Qwen3.8-Flash-Next in 32 GB because only the experts are streamed, the 51 GB n-gram table stays entirely in memory. Ziadé's project solved this by reading the table from disk on demand too, a detail that shows how each MoE model can hide a different bottleneck than the obvious one.

Translated from the Brazilian Portuguese original · Read the original

View profile →