[apexalpha]

I bought two (relatively) old datacenter GPUs with 48gb VRAM total for €200 that gets me 7 token/s for a 70b model.

[383toast]

which GPUs?

[zargon]

Not the GP, but I bought a few P40s over the summer for $150 each. Last I checked they're more expensive now, but it's still cheap vram and fast enough at inference for me.

[apexalpha]

Nvidia M40 and P40.