DeepSeek releases inference stack capable of 155 trillion tokens per day with margins up to 80%

DeepSeek published, on October 20, 2026 (coinciding with the first day of PyTorch Conf), DeepInfer: the inference engine that runs in production behind the V4.1 Flash model. Along with it came a detailed engineering report, translated and analyzed by the Latent Space newsletter, with throughput and cost numbers that the company rarely discloses at this level of detail.

The package is a customization of vLLM, with explicit credit to SGLang and TensorRT-LLM, released under the Apache 2.0 license. It came with deepseek-recipe, a component that handles tokenization, decoding, and compatibility with the Chat Completions, Responses, and Messages APIs.

What was opened, and what was left out

DeepSeek itself calls the release a complete open source package, but the technical report admits the opposite on at least two central points. KV cache management and load balancing across workers, the pieces the community has already nicknamed the "crown jewels," remain outside the repository because they're tied to the company's internal infrastructure.

This changes what the release actually represents. It's not DeepSeek's production service running outside its own walls: it's the execution engine, without the persistent cache system and without the orchestrator that distributes load across hundreds of GPUs. Whoever clones the repository gets a solid base on top of vLLM, not a 1:1 copy of the announced economics.

The numbers behind the headline

The report brings a combination of throughput and pricing that, added together, arrives at the reference figure: about $22 million per day in revenue per cluster, calculated from 155 trillion tokens/day and the current price of $0.15/$0.60 per million input/output tokens. Other numbers from the same report:

• 96% input token cache hit rate
• 400 ms average time to first token
• 230 tokens/s generation, reaching 300 tokens/s on coding tasks
• $38 in revenue per accelerator-hour during low-demand periods

From these numbers, the technical community tried to reconstruct the profit margin, and here the math gets unstable depending on the GPU cost assumption:

The first estimate started from an ideal scenario projecting about $49 per hour in revenue; when confronted with the actual number disclosed by DeepSeek ($38/hour), the math was revised to 84.2% to 96.1% margin. In none of these cases do the figures include cache SSD storage costs, networking, idle capacity, engineering, and support: these are contribution margins on hardware, not the business's net margin.

Separating prefill, decode, and cache changes the cluster design

The disclosed architecture separates visual encoding, prefill, and decode into stages that scale independently. The deployment recommendation is to use common eight-accelerator machines for prefill and supernodes of 64 to 128 accelerators for decode, with expert parallelism distributing MoE blocks across large batches.

Some scheduling details explain why comparing "tokens per batch" across different engines tends to be misleading:

1. Prefill scheduling estimates the computational cost of attention, not the raw size of the request, and varies the chunk size to fit within a time budget.
2. Prefill routing doesn't require cache affinity: since prefixes live in shared storage, any available worker can handle the request.
3. Decode routing balances both the number of running requests and the accumulated size of the KV cache.
4. Popular experts are periodically replicated to reduce inter-device waiting caused by routing imbalance.

One detail caught the attention of inference engineers: the disclosed Blackwell configuration apparently doesn't use tensor parallelism, something unusual in standard vLLM and SGLang recipes. The community's reading links this to interconnect constraints from V3's H800 era, but it's an interpretation of a specific configuration, not a general recommendation against tensor parallelism.

The cache is the piece that sustains the 155 trillion

The persistent KV cache lives in a separate storage cluster (3FS), with about 100 PiB in SSD. Retention is three days for agent traffic and seven days for chatbot traffic, and any prefill worker can fetch a cached prefix, decoupling cache locality from the physical position of computation.

This matters for anyone building agents that repeatedly resend a growing history of instructions, repository content, and tool outputs. It also means that the 155 trillion tokens served per day aren't 155 trillion freshly computed tokens: a large share is cache hits, and the 96% hit rate cited above is the data point that makes the throughput number understandable.

A technique called SWA Bounded Replay reconstructs sliding-window attention state using only the last 128 tokens, instead of redoing the calculation over the entire previous history. DeepSeek reports that this nearly halves total prefill computation, an approximation that works for their architecture, not a guarantee of portability to any model.

The timing of the announcement also invites a direct comparison: OpenAI had just announced a 12-hour prompt cache (up from a 30-minute default), while DeepInfer sustains cache windows of up to 72 hours. These are different products and architectures, but both target the same problem: agents that reuse context after long intervals.

What vLLM has already proven, without needing the missing piece

The most concrete answer to the question "can only DeepSeek pull this off?" came from the vLLM team itself. After noting that DeepInfer was originally built on top of vLLM, with explicit credit and an Apache 2.0 license, the project published results showing decode throughput within 6% of the numbers disclosed by DeepSeek, on GB200, under an equivalent configuration.

It's an important result, but a limited one: parity at the decode stage isn't reproduction of the entire service. There's no equivalent, in vLLM's tests, to the 100 PiB of cache, the same retention policy, the same routing, or the same availability under DeepSeek's real traffic. Woosuk Kwon himself, vLLM's creator, described DeepSeek as a long-time adopter and contributor to the project, which repositions DeepInfer: it's production knowledge flowing back into an ecosystem that already existed, not a complete substitute for it.

NVIDIA, Ascend, and the hardware gap that remains open

DeepInfer runs on both NVIDIA (Blackwell/SM100) and Huawei Ascend, but a later hardware analysis compared decode efficiency between GB200 and Ascend 950DT: with a 100 tokens/s constraint and 131K context, the difference lands around 25,500 versus 5,000 tokens/s per device; at 230 tokens/s, the ratio climbs from about 5x to 7.4x in NVIDIA's favor. Compatibility with Ascend isn't synonymous with equivalent production efficiency.

To gauge the size of the investment required, a community estimate (from Teortaxes, with the explicit caveat that it's a proposed scenario, not a proven reproduction) calculates that matching the throughput of DeepSeek's cluster would require somewhere between 3,000 and 4,000 GB200s.

What changes for those running inference outside DeepSeek's scale

For teams operating their own models or hosting inference at a smaller scale, the practical value of the release lies in the architectural patterns, not in the margin numbers. Separating prefill and decode into pools with different hardware profiles, routing by estimated computational load instead of raw token count, and using adaptive speculative decoding (DeepSeek's DSpark reports a 6% throughput gain over always verifying five draft tokens) are portable ideas even without a cluster of thousands of GPUs.

Since vLLM already reproduces much of DeepInfer's decode behavior and is Apache 2.0, that's the more realistic path for most teams: adapting the same scheduling ideas within an engine that already has community support and maintenance behind it, instead of trying to rebuild the part DeepSeek chose not to open.

The 80% margin and the 81-day payback period depend on traffic volume, cache hit rate, and cluster investment that few operations outside DeepSeek will reach; trying to replicate the number without matching scale is the fastest way to burn through GPU budget instead of saving it.