How Cloudflare serves Kimi and GLM at scale with FP8 and INT4

Cloudflare wrote up how it runs two large open-weight models, Moonshot's Kimi and Z.ai's GLM, cheaply enough to serve at scale, and the post is unusually concrete about the tradeoffs. The core moves are two forms of compression plus a safety check. For Kimi, the team quantized the attention KV cache from 16-bit to 8-bit floating point, which halves its size and lets a single deployment hold about 1.37 million tokens instead of 686,000. That capacity translates directly into concurrency: 64 simultaneous requests instead of 32, about 41% higher throughput, and roughly 30% lower cost per token.

For GLM 5.2 they went further and compressed the weights themselves from 8-bit to 4-bit integers, shrinking the checkpoint from 705 GB to 421 GB and cutting per-GPU memory from 88 GB to 52 GB. Decoding got 55% faster at low concurrency. The third piece is a KV cache integrity check that validates page mappings before each decode step so a request cannot read another request's memory, and it costs under 1% in throughput. Across GSM8K, MMLU, and ARC, accuracy moved by less than 0.8 points, which is the number that makes the rest usable.

The full engineering notes are on the Cloudflare blog. The takeaway is that the gap between a frontier open model and a servable one is now mostly a quantization and memory problem, not a capability problem.

Why it matters

If you are deciding whether to self-host an open model or pay for a hosted API, these numbers set the reference: FP8 KV cache and INT4 weights can roughly halve your serving cost while barely touching accuracy. The catch is engineering time, so the question is whether your volume justifies building what Cloudflare just described.