[rini17]

Note that quantized versions of llama3 70B can be ran on CPU on much cheaper server. I am personally using it via llama.cpp on bare metal 6-core Xeon CPU with 128G RAM for ~50 euro monthly.

[JJneid]

Is inference speed an issue for you?

[rini17]

Sufficient for fluent conversation.

[JJneid]

usually performance takes a hit with quantization. are you getting quality responses?

[rini17]

Since llama3, yes, quite satisfying.