[cypress66]

The performance loss is because this is RTN quantization I believe. If you use the "4chan version" that is 4bit GPTQ, the performance loss from quantization should be very small.

[xdennis]

What's the 4chan version?

[aseipp]

See https://github.com/ggerganov/llama.cpp/issues/62 (the related repo was originally posted on 4chan, is all, but the code is on GitHub)

[cypress66]

https://rentry.org/llama-tard-v2