|
|
|
|
|
|
by meatmanek
9 days ago
|
|
|
There are 1-bit* LLMs (+1/-1, usually), though they often come with an additional scaling parameter: https://huggingface.co/prism-ml/Bonsai-27B-gguf#weight-repre... > Weight Representation: Q1_0_g128
Each weight is a single sign bit: 0 maps to −scale, 1 maps to +scale. Every group of 128 weights shares one FP16 scale factor.
> Effective bits per weight: 1.125 (1 sign bit + 16-bit scale amortized over 128 weights) The Bonsai ternary model also uses this group-wise scaling, so the theoretical bits per parameter is 1.71 instead of ~1.58; additionally, they actually store trits as 2 bits; with the group scaling factor it becomes 2.125 bits/weight: https://huggingface.co/prism-ml/Ternary-Bonsai-27B-gguf#memo... |
|
|