Slotstream runs a 104 GB model on a 48 GB Mac by streaming from SSD

Running a language model that takes up 104 GB on disk on a Mac with only 48 GB of RAM has always hit the same wall: either the weights fit in memory, or you give up.

Editor's note: we were unable to confirm the full name of the project's author. The available source (the README and GitHub page) identifies only the user 'carloslfu', without revealing a legal name. We chose to keep only this verifiable identification in the text; this does not affect the project's technical numbers and behaviors, which were checked line by line against the original documentation.

Slotstream, an open source project by the developer identified on GitHub as carloslfu, published on Hacker News, proposes a different path: keep the bulk of the model on the SSD and stream to RAM only the parts each token needs. The result measured by the author is Qwen3.8-Flash-Next (125B parameters, MoE architecture, 104 GB in 4-bit) running at about 12 tokens/s on a 48 GB MacBook M5 Pro.

For those building software in Brazil and looking at local AI, the point isn't the number itself, but what it unlocks: models that used to require a workstation with 128 GB (or an expensive dedicated GPU) become viable on hardware that a lot of people already have on their desk.

Why traditional mmap doesn't solve it

The trick almost everyone would try first, mapping the file into memory with mmap and letting the system page it in, doesn't work here, and the README explains why. MLX (Apple's machine learning framework for Apple Silicon) cannot materialize just a part of a memory-mapped tensor. According to the author:

MLX cannot materialize part of a memory-mapped tensor: a top-10 expert gather evaluates all 512 experts of that layer, and a 16-row n-gram lookup evaluates the whole 250 MB shard, so an mmap path loads ~100 GB and dies.

>

-- slotstream README

In practice, the standard mlx_lm.load() route drove the 48 GB Mac to 48 GB of swap without generating a single token. The mmap loads everything and dies.

How slotstream distributes the 104 GB

The trick lies in understanding where the model's bytes are. In an MoE (Mixture of Experts), most of the weights aren't used at each step. The distribution of Qwen3.8-Flash-Next, according to the project:

| Component | Size | Behavior | |---|---|---| | Routed experts | 68 GB (512 per layer, 10 active per token) | Read from SSD on demand | | N-gram table | 32 GB | Read from SSD (250 MB shards) | | Dense trunk | 3.8 GB | Stays resident in RAM |

Experts are read with pread directly into a fixed pool of cache slots shared across the 48 layers, so that "hot" layers borrow slots from "cold" ones. The dense trunk, which is tiny compared to the rest, stays in memory the whole time.

An important detail for anyone concerned with reproducibility: cache size changes the speed, never the output. Greedy decoding is byte-for-byte identical between a 4 GB cache and a 24 GB one, and this is guaranteed by tests in the project.

What to expect by memory tier

The project self-sizes: with no flags, it reads the machine and picks a target. Only the 48 GB tier was measured on real hardware (the aforementioned M5 Pro); the rest are derived from the same curve and come with the caveat that smaller Macs also have slower SSDs.

| Memory | Estimate | |---|---| | 8 GB | below the 8.1 GB floor; doctor warns it will page | | 16 GB | ~5 tok/s (estimated) | | 24 GB | ~8 tok/s (estimated) | | 32 GB | ~10 tok/s (estimated) | | 48 GB or more | ~12 tok/s (measured); caps at 33 GB, the rest of the machine stays free |

The 33 GB figure isn't a kindness, it's the "knee" of the curve: the smallest target where the expert cache beats the decode plateau and there's still budget left for fast 4,096-token prefill. According to the author, sweeping GB by GB, nothing between 34 and 84 GB improves speed. In other words, a 64 GB or 128 GB Mac asks for the same 33 GB, the surplus buys nothing. The process is also elastic: it re-evaluates every 15 s and resizes the cache between requests.

Disk is the real gatekeeper

Before RAM, what actually blocks you is the SSD. ~110 GB free are needed, which makes a 512 GB Mac the realistic minimum, no matter how much memory it has. The weights download is one-time: 103.8 GB across 24 files, verified against sha256 hashes compiled into the binary (a truncated or corrupted download never reaches the engine). Interrupting it is safe, it resumes at the exact byte.

A curious finding from the source: the download bottleneck is Hugging Face, not your connection. Above ~400 Mbps, more bandwidth doesn't help, the fast hf_xet client itself stalled in the 36 to 57 MB/s range while the same link did 134 MB/s to a regular host.

Installation and Ollama-compatible API

Installation is one line (Apple Silicon and macOS 14+):

curl -fsSL https://raw.githubusercontent.com/carloslfu/slotstream/main/install.sh | sh

Or build it from scratch, no Xcode needed (Command Line Tools are enough):

git clone https://github.com/carloslfu/slotstream && cd slotstream
make build

The point that matters most for anyone with code already pointing at local models: serve listens on port 11434 and implements the chat/generate subset used by Ollama clients and OpenAI SDKs. In other words, you can plug it in without rewriting the integration:

curl localhost:11434/api/chat -d '{
  "model": "qwen3.8-flash-next:4bit",
  "messages": [{"role": "user", "content": "hello"}]
}'

Streaming, CORS, and the usual sampling options (temperature, top_p, top_k, min_p, presence_penalty, seed, num_predict, stop) are supported. What isn't (tools, images, JSON-schema output, logprobs) returns a clear 400 instead of being silently ignored. Open WebUI, the Ollama CLI, and the OpenAI SDKs were tested against this subset.

Where it still hurts

The project is honest about its limits. Long prompts are the slow axis: everything in the prompt is processed before the first token. Prefill runs at ~50 tok/s on a 16 GB Mac and ~125 on a 48 GB one, so an 8,000-token prompt waits roughly one to three minutes for the first token. Within a conversation this is paid once, subsequent turns only reprocess what's new (measured: 6.0 s instead of scaling up to 25.8 s over eight turns). The total context (prompt plus response) is capped at 32,768 tokens.

Another point: full expert residency (all 512 per layer, with no SSD reads) would require about 88 GB and has never been measured. And the runtime has only been exercised on one machine, the 48 GB M5 Pro, the other tiers are curve extrapolation. On macOS 14 and 15 only the installer was tested, not the runtime.

The license is MIT (the weights follow Qwen's community license). PLAN.md has the design and the milestone tracker, and MEASUREMENTS.md documents every number along with the method, including the experiments that failed, which is a good sign of rigor for anyone evaluating whether it's worth bringing into their own setup.