[jasonjmcghee]

Depends how much ram yours has. Get a 4bit quant and it'll fit in ~40-50GB depending on context window.

And it'll run at like 40t/s depending on which one you have