Limits of Language Model in Triple Bit System

Artificial intelligence researchers developed a new technique to reduce memory consumption of language models. Ternary Large Language Models (LLM), each weight is only stored with three symbols (-1, 0, +1), providing significant savings compared to traditional systems. However, this approach’s long-term discussion has been in practice.

The standard approaches that are valid until now were:

• Theoretical limit calculated approximately 1.585 bit/weight (log2 3 = 1.585)
• Five ternary weight in practice fit into one bytes 1.625 bit/weight effective
• Due to the group sizes with the force of two, this rate was higher

The new research has managed to overcome this 1.625-bit barrier and made models lighter. This development can revolutionize in organizations that want to reduce server costs especially on devices with limited resources. Ternary systems can maintain almost the same performance while consuming less memory and energy.

This finding opens the way AI models become more accessible and sustainable. The industry sees critical to expand large language models of such efficiency improvements.