[rmbyrro]

Do you have a ballpark idea of how much RAM would be necessary to run llama 3.1 8b and 70b on 8-quant?

[karolist]

Roughly, at Q8 the model sizes translate to GB, so ~3 and ~70GB.

[ekianjo]

You mean 8, not 3?

[karolist]

Yes, apologies, can't edit now