Cloudflare details flaw that leaked data between GLM-5.3 model users

What happened

Between August 27 and September 9, 2026, the GLM-5.3 model (@cf/zai-org/glm-5.3), available on Cloudflare's Workers AI, returned to some customers fragments of responses that belonged to other users. Cloudflare confirmed, in a report published on October 26, that 155 responses were affected during that period, all generated from prompts with more than 681,280 tokens.

The case came to light on September 8, when security researcher Gadi Evron, from the company Knostic, reported the issue through Cloudflare's bug bounty program on HackerOne. The team reproduced the leak the following morning and took the model offline 40 minutes after declaring an incident. The service was restored the same day, with a corrected configuration and a patched version of the SGLang inference engine.

According to Cloudflare, the volume and timing of the affected requests are consistent with legitimate long-context usage, not deliberate exploitation of the flaw. No other model hosted on Workers AI was affected.

How the leak happened

GLM-5.3 is a Z.ai model aimed at agentic coding workflows, run by Cloudflare on a disaggregated prefill/decode architecture: a pool of prefill GPUs processes the prompt and generates the KV cache, which is transferred via RDMA (using the Mooncake framework) to a pool of decode GPUs, responsible for generating the response. The KV cache lives in a shared memory area, organized into fixed-size pages reused across requests without zeroing: the next request that receives a page simply overwrites whatever was there.

Two problems combined to cause the leak:

1. Incompatible configuration. Cloudflare published GLM-5.3 with a context window of 1,048,576 tokens, but the KV cache pool on the prefill GPUs only had room for 681,280 tokens. The team knew about the gap, but pre-launch tests showed correct HTTP status codes and error-free streaming even in long contexts, which led the team to believe the engine handled the difference well.
2. SGLang bug. When a prompt didn't fit in the pool, the scheduler was supposed to reject the request. Instead, it swapped the prompt for a one-token placeholder and let the request continue through the normal batch-processing flow, without actually aborting. On a single node this caused no problem, because the client received the error at the end. In the disaggregated mode Cloudflare uses, the prefill and decode admission queues only checked whether the request was finished(), and a rejection was stored in to_finish, not in finished_reason, so that check kept returning false.

In practice, decode reserved memory pages for the entire prompt, but prefill only wrote one page (the placeholder's) and reported "transfer complete." Decode then generated the response by reading all the reserved pages, including ones that still held data from earlier requests by other users. The client received HTTP 200, with prompt_tokens matching the size of the original prompt and coherent text, only partially leaked from another session.

Timeline

What Cloudflare fixed

In the configuration, GLM-5.3's KV cache pool was increased to about 2.04 million tokens, nearly double the 1,048,576-token context window. This eliminates the rejection path for any prompt that fits within the advertised window, which by itself already prevents the bug from being triggered.

In the engine, Cloudflare proposed to the SGLang project patch #38704, which causes a request marked for rejection to be terminated in the disaggregation admission queue before any memory allocation or KV cache transfer. Part of this logic was already in the process of being merged as #36988, for a related failure path, and the maintainers accepted Cloudflare's approach in #38935.

In addition, the company extended deferred page release to also cover source pages on prefill, limited the size of transfer batches via Mooncake, and changed the load balancer to no longer abort a request mid-transfer when the client disconnects. All of these changes already apply to every model Cloudflare runs on SGLang, not just GLM-5.3.

Why this matters for teams running AI in production

The case is relevant to any team that operates its own inference with SGLang in disaggregated prefill/decode mode, not just Workers AI users. Cloudflare published direct recommendations for these teams:

• Measure the size of the KV cache pool and compare it against the context window, on every prefill and decode node, with every change to weights, quantization, parallelism, or memory fraction.
• Update to SGLang v0.5.22 or later, which already includes patches #36988 and #38935.
• Run with deferred KV cache release enabled in both roles (prefill and decode), the default since PR #41023, and limit the size of transfer batches via Mooncake.
• Enable the --disaggregation-enable-kv-checksum and --kv-canary flags when the model architecture supports them, to detect cache corruption.
• Configure the proxy to avoid aborting requests midway, letting them proceed at least until the first token.

Worth noting: not every model has support for integrity checks today. Cloudflare itself cites GLM-5.3, GLM-5.3 Flash, and DeepSeek V4 as architectures not yet covered by KV cache checksumming, which means teams running these models still lack an automatic safety net against this kind of corruption.

What remains open

Cloudflare is direct in stating that the configuration and memory lifecycle fixes reduce the chance of this kind of bug, but don't eliminate it entirely. The report frames the incident as part of a broader class of KV cache memory bugs the company has been addressing for months, including races between nodes (when one node releases a page while another is still writing to it) and page lifetime issues within the same node.

For these cases, the long-term plan involves integrity checks at every generation step, not just during the prefill-to-decode transfer, work that Cloudflare describes as still ongoing with the SGLang maintainers.