BITCOS Breakthrough: New Storage Layout Shrinks Ternary LLMs Below 1.58 Bits

Researchers have introduced BITCOS, a new weight-storage layout for ternary large language models (LLMs) that breaks the conventional 1.58-bit storage barrier by exploiting the significant sparsity found in these models. This innovation achieves more compact storage and improves inference efficiency on platforms like Intel Xe2 GPUs.

Addressing the Ternary LLM Storage Challenge

Ternary LLMs, which use weights represented by -1, 0, or 1, traditionally store each weight at approximately 1.585 bits. This figure derives from log2(3), reflecting the three possible states. While ternary quantization offers benefits in reducing model size and computational demands compared to full-precision models, the storage efficiency still presented an optimization opportunity.

The research team identified a critical insight: a substantial portion of weights in ternary LLMs are zero. Their analysis of 29 different ternary LLMs revealed that zeros can constitute up to 51.5% of all weights. This high degree of sparsity became the foundation for BITCOS's design.

How BITCOS Achieves Greater Efficiency

BITCOS exploits this sparsity through a clever combination of a dense presence bitmap and a compacted sign vector. Instead of storing each of the three possible values (-1, 0, 1) directly, BITCOS uses a bitmap to indicate whether a weight is zero or non-zero. For non-zero weights, a separate sign vector then specifies if the value is -1 or 1. This approach allows for a more efficient representation, costing 2-z bits per weight, where 'z' represents the zero density.

This method enables BITCOS to store weights more compactly than standard five-trit packing, a common technique for ternary weight storage. In fact, BITCOS outperformed five-trit packing in 26 out of the 29 tested models, achieving an impressive 1.485 bits per weight on the sparsest model. This represents a tangible improvement over the previous 1.58-bit standard.

Performance Gains in LLM Inference

The practical benefits of BITCOS extend beyond just storage. The more compact weight representation translates directly into performance improvements during LLM inference. End-to-end LLM inference tests demonstrated significant decode throughput enhancements:

• Up to 1.18x improvement on CPUs
• Up to 1.27x improvement on GPUs

These gains are further amplified by optimized unpacking sequences specifically designed for mainstream hardware architectures, including AVX-512, AVX2, and Intel Xe2 GPUs. This ensures that the benefits of the compact storage are fully realized during computation, leading to faster processing and potentially lower operational costs for AI news applications and other demanding workloads.

Why This Matters for AI Development

The introduction of BITCOS is a notable step forward in the ongoing effort to make large language models more efficient and accessible. By reducing the storage requirements and improving inference speed, this technology can:

• Lower operational costs: Less memory usage and faster processing can lead to reduced infrastructure expenses for deploying and running LLMs.
• Enable broader deployment: More efficient models can be deployed on a wider range of hardware, including devices with more constrained resources.
• Accelerate research and development: Faster inference cycles allow researchers and developers to iterate more quickly on new models and applications.
• Enhance user experience: Quicker response times from LLMs can lead to a more seamless and responsive user experience in various AI-powered tools.

This innovation underscores the continuous advancements in optimizing AI models, moving towards a future where powerful LLMs are not only more capable but also more resource-friendly.

Conclusion: What's Next for Efficient LLMs

BITCOS represents a significant stride in optimizing ternary LLMs by intelligently exploiting weight sparsity. The demonstrated improvements in storage efficiency and inference throughput on both CPUs and GPUs highlight its potential to impact the deployment and performance of AI applications. As the demand for efficient and powerful AI tools grows, innovations like BITCOS will be crucial in shaping the next generation of artificial intelligence. Developers and researchers will be watching to see how these techniques are integrated into broader AI platforms and frameworks, potentially influencing the pricing and accessibility of advanced AI capabilities.

Sources

• [2609.16338] Breaking the 1.58-bit Barrier for Ternary LLMs
• [RFC] Introduce a Host Weight Runtime for reusable runtime-ready artifacts · Issue #6414 · vllm-project/vllm-omni · GitHub
• bips/bip-0141.mediawiki at master · bitcoin/bips - GitHub
• [RFC] Fail-Closed Graph Storage Contract for Weight Reload · Issue #48478 · vllm-project/vllm · GitHub