155 answers carried another customer's data

Cloudflare has published a disclosure of a tenant-isolation failure in GLM-5.3, one of the models it hosts on Workers AI. Between 27 August and 9 September a small number of customers received responses that contained data from other customers' requests made to the same model shortly before theirs. In total 155 responses were affected, all of them to prompts longer than 681,280 tokens. The issue was reported on 8 September by Gadi Evron of Knostic through Cloudflare's HackerOne programme; the company reproduced it the next morning, took the model offline within 40 minutes of declaring an incident and restored it the same day. It says no other Workers AI model was affected and that it found no evidence of deliberate exploitation.

Two problems overlapped. The first was configuration. The model was deployed with a 1,048,576-token context window, while the KV cache pool on the prefill GPUs had room for only 681,280 tokens. Cloudflare says it knew about the mismatch before launch and expected the engine to return an error for any prompt it could not hold; its long-context tests passed, so it did not treat the gap as a problem.

The second was a bug in SGLang, the open-source inference engine, in disaggregated prefill and decode mode. An over-length prompt was marked as rejected and replaced with a one-token placeholder, but instead of aborting, prefill built the cache for the placeholder and sent it to the decode node. Told that the transfer was complete, the decode node assumed the cache had been overwritten for the new request, when most of the previous state was still there, and generated an answer while attending over stale pages that held earlier users' data.

The fixes address both. The pool was resized to hold about 2.04 million tokens, roughly twice the context length, so a valid prompt can no longer reach the rejection path. The engine was patched so that a request carrying a pending rejection is retired with its original error before any allocation or transfer, and the decode node now checks the number of tokens it received against the number it reserved; the changes went upstream as part of a coordinated disclosure with SGLang. Cloudflare's own conclusion is the one worth keeping: relying on pool size alone is fragile, because quantisation, parallelism or a new model version can change it in ways that are easy to miss.