The most damaging inference bug of the year never returned an error. For thirteen days in late August and early September, Cloudflare's hosted copy of GLM-5.3 answered prompts longer than 681,280 tokens with HTTP 200, a coherent reply, and a prompt_tokens count that matched what the client sent. Some of those replies also contained fragments of other customers' prompts. The postmortem, published October 26 in coordination with the SGLang maintainers at RadixArk, is worth reading closely because the failure mode is not exotic. It is a rejected request that kept reporting itself as alive.
The Configuration Gap That Set the Trap
Workers AI serves GLM-5.3 with SGLang in a disaggregated layout. A pool of prefill GPUs computes the KV cache for an incoming prompt and hands it to a pool of decode GPUs over RDMA using Mooncake. The model was deployed with a 1,048,576-token advertised context window, but the KV pool on the prefill nodes only held 681,280 tokens. SGLang sizes that pool at startup from leftover GPU memory, so the two numbers are independent config values that nothing forces to agree.
Cloudflare knew about the mismatch before launch. Their long-context tests returned successful responses, so the team assumed the engine handled over-length prompts gracefully. That assumption held on single-node deployments, where a rejected prompt was swapped for a one-token placeholder, the batch drained, and the client got a 400. The assumption did not survive disaggregation.
On September 8 at 23:11 UTC, Gadi Evron of Knostic filed a HackerOne report: very long prompts on @cf/zai-org/glm-5.3 sometimes returned text from other sessions. Cloudflare reproduced it with two independent accounts in 25 minutes, declared an incident at 13:22 the next day, pulled the model at 14:00, rolled corrected config to all prefill nodes by 14:52, and restored public access at 19:40. In total, 155 responses were affected. Every one of them followed the same mechanical path.
Anatomy of a Cross-Tenant Attention Read
The KV cache in GPU memory is a shared pool of fixed-size pages. Pages are assigned when a request is admitted and released when it finishes, but they are never zeroed in between. The next request simply overwrites what is there. That is a deliberate performance choice across inference engines, and it makes page ownership tracking the engine's entire responsibility.
Here is the sequence that leaked data, in the order the engine executed it:
- Bootstrap. The router told the decode node the prompt length, and decode reserved pages for it, eight in Cloudflare's reproduction.
- Prefill admission. The admission queue admitted any request where
req.finished()returned False. A rejected request stored its rejection into_finish, notfinished_reason, sofinished()still returned False and the dead request walked in. - Prefill forward pass. Rejection had already replaced the long prompt with a one-token placeholder, so prefill ran a trivial pass on a single token.
- KV transfer. Prefill shipped one page of KV to decode and signaled transfer complete.
- Decode admission. The same
finished()check let the handoff through. - Decode attention. Decode generated a response while attending over all reserved pages. Only the first had been written by this request. The rest still held other users' prompts and partial generations.
The client saw nothing wrong. The reply was fast, coherent, and correctly counted. The tell was diagnostic only in hindsight: an eight-page prefill should not finish that quickly. The bug had already been filed to SGLang as issue #30233 on July 6, 2026, and Cloudflare only noticed that report while investigating its own incident.
Two Fixes, Because One Is Not Enough
Cloudflare shipped a configuration change and an engine patch. The config change brought the prefill pool to roughly 2.04 million tokens, about twice the context length, so a valid prompt always fits and the rejection path is never reached. The engine patch retires requests carrying a pending rejection at the disaggregation admission queue, before any bootstrap, allocation, or KV transfer, and makes the decode node verify that the transferred token count matches what it reserved. Cloudflare proposed that as #38704; upstream landed the equivalent in #36988 for v0.5.20 and #38935 for v0.5.21.
The config fix alone was rejected internally as fragile, and the reasoning is the useful part for anyone operating disaggregated serving. Pool size shifts with quantization, tensor parallelism, CUDA graph sizes, and model version, and none of those changes announce themselves. Every other model Cloudflare serves happened to have a pool larger than its context window, which is the only reason GLM-5.3 was the sole affected deployment.
With the exposure closed, the remaining work was a class of bugs rather than a single one. SGLang v0.5.22, released October 19, carries per-transfer checksums, abort acknowledgement preservation, deferred KV release on by default, and cancellation of both request halves on client disconnect or prefill failure. Cloudflare now gates every disaggregated deployment change on four measured checks: pool size against context length, deferred release on both roles, transfer batch bounds inside the release timeout, and all required patches present.
Independent confirmation arrived from Seoul National University's CompSec Lab, which built a KV cache sanitizer called KVGuard. On SGLang 0.5.19 it flagged 4,095 uninitialized reads on the leaking request while normal traffic stayed silent. The lesson generalizes past one vendor: as long as inference engines recycle GPU memory without clearing it, tenant isolation is a property of the scheduler, not of the model weights.
Comments