Content deleted Content added
→top: Expanding article |
→top: Copyedit (minor) |
||
Line 1:
{{in use}}
A '''1.58-bit Large Language Model''' ('''1.58-bit LLM''') is a version of a [[large language model]] with weights using only three values: -1, 0, and +1. This restriction allows the model to replace costly multiplications with additions and reduce the storage memory. Since the end-task performance and [[Perplexity (LLM)|perplexity]] of the 1.58-bit LLMs are close to their "full precision" (16-bit [[FP16]] or [[BF16]]) counterparts, this design allows reaching the same [[artificial intelligence]] goals with much lower hardware requirements, latency, and training effort.{{sfn|Ma|Wang|Ma|Wang|2024|p=1}}
The name comes from a fact that a single [[Ternary numeral system|trit]], a [[ternary arithmetic]] equivalent of a bit that can take the {-1, 0, 1} values, carries <math>log_2 3 \approx 1.58</math> [[bits of information]].
| |||