[chriscappuccio]

Better to run the Q8 model on an epyc pair with 768GB, you'll get the same performance

[ltbarcly3]

The Q8 model is totally different?

[manmal]

My experience with quantizations is that anything below 6 is noticeably worse. Coherence suffers. I’ve rarely gotten anything really useful out of a Q4 model, code wise. For transformations they are great though, eg convert JSON to Markdown and vice versa.

[ltbarcly3]

No I mean the quantized versions of this model in particular have less parameters as well. They are almost different models.

[yieldcrv]

I like Q5

The sweet spot for me