Cloudflare Boosts Kimi & GLM AI Model Performance with FP8 Quantization

Cloudflare's Workers AI platform now efficiently serves large Chinese frontier models like Moonshot's Kimi K-series and Z.ai's GLM by implementing KV cache quantization, a technical deep-dive published on August 3 details. This innovation addresses a critical infrastructure challenge by enabling the deployment of increasingly large frontier models on constrained GPU memory.

Optimizing AI Inference with FP8 Quantization

The core of Cloudflare's optimization lies in quantizing the Key-Value (KV) cache from 16-bit Brain Floating Point (BF16) to 8-bit Floating Point (FP8). This technical shift is crucial for managing the substantial memory demands of large language models. By reducing the precision of the KV cache, Cloudflare effectively halves memory usage, which in turn doubles the context window capacity available to the models.

This improvement is particularly evident with models like Kimi K2.6. Running on H200 GPUs, the FP8 quantization increased the maximum context window from approximately 686,000 tokens to an impressive 1.37 million tokens. This expanded context window allows AI models to process and understand significantly longer inputs and maintain more extensive conversational histories, leading to more coherent and capable interactions.

Enhanced Throughput and Cost Efficiency

Beyond memory optimization, FP8 quantization also delivers substantial performance and cost benefits. For Kimi K2.6, the change enabled the system to handle 64 concurrent requests, a significant increase from the 32 requests possible with BF16. This led to a throughput of 2,192 tokens per second, resulting in approximately 41% higher overall throughput and a 30% reduction in cost per token. These efficiencies are vital for making advanced AI models more accessible and scalable for developers and businesses utilizing the Workers AI platform.

Leveraging SGLang for Superior Inference Serving

Cloudflare employs SGLang as its primary inference serving framework, recognizing it as the top-performing open-source option available. SGLang plays a critical role in managing the complex demands of serving large-scale AI models, ensuring stable and efficient operations. Cloudflare actively contributes to the SGLang open-source project, upstreaming patches and new features, which benefits the broader AI community.

In addition to KV cache quantization, Cloudflare also applies other optimization techniques, including model weight compression and shared-cache integrity protections. These combined efforts ensure robust and efficient operation of frontier models, demonstrating that advanced Chinese models can be served effectively alongside Western alternatives on a global scale.

Why This Matters for AI Development

The ability to efficiently serve increasingly large frontier models on constrained GPU memory is a persistent and critical infrastructure challenge in the AI landscape. Cloudflare's advancements with FP8 quantization and its strategic use of SGLang offer a practical solution to this problem. This development not only makes powerful models like Kimi and GLM more accessible but also sets a precedent for how other platforms might approach similar scaling challenges. For developers, this means access to more capable and cost-effective AI tools, fostering innovation across various applications. Learn more about latest AI updates and AI platforms.

Key Takeaways

• Cloudflare's Workers AI platform now efficiently serves large Chinese frontier models like Kimi and GLM.
• FP8 quantization of the KV cache halves memory usage and doubles context window capacity.
• Kimi K2.6 on H200 GPUs can now handle 1.37 million tokens and 64 concurrent requests.
• This optimization leads to 41% higher throughput and 30% lower cost per token.
• Cloudflare utilizes and contributes to the SGLang open-source inference framework.

Sources

• GitHub - sinameraji/kimiflare: Moonshot kimi k3 terminal based coding agent & harness running on your own Cloudflare account. · GitHub
• feat: add reasoning_content, GLM, GPT-OSS, Nemotron, Kimi K2.5 models · cloudflare/langchain-cloudflare@f08d4e8 · GitHub
• "No need for cookie banners" might be incorrect · plausible/analytics · Discussion #1963 · GitHub
• frontend-case-studies/readme.md at master · andrew--r/frontend-case-studies · GitHub