|
|
|
|
|
|
by SushiHippie
787 days ago
|
|
|
* ~3GB with 8bit quantization. Without quantization it is ~6GB [0]. 8 bits = 1 byte 3 billion * 1 byte = 3 gigabyte + Some memory for the context of the LLM [0] 3b-instruct has a total file size of 4.94GB + 1.13GB which is 6.07GB which can be seen here: https://huggingface.co/apple/OpenELM-3B-Instruct/tree/main A bit of overhead will always be there, as you probably want to store some metadata next to the raw weights. |
|
|