[broadsidepicnic]

what you maybe forget here is the use case for people and businesses who can not send the data to 3rd party due to privacy/contractual reasons. This is what I'm looking at, we're bound by strict policies for data sharing outside of our premises.

[djx22]

yes, I understand the usecase. Where I'm coming from is quantized vs. unquantized. 4bit quants are lobotomizing the model heavily to the point that it's better to invest in some capable hardware than keep fighting the limitation. Refurbished server grade hardware is accessible. For the price of an RTX 6000 PRO you could probably get much more VRAM but 1-2 generations older.