[trvz]

To start with, an Epyc server or Mac Studio with 512GB RAM.

[criddell]

I looked up the price of the Mac Studio: $9500. That's actually a lot less than I was expecting...

I'm guessing an Epyc machine is even less.

[graeme]

How does the mac studio load the trillion parameter model?

[petu]

By using ~3 bit quantized model with llama.cpp, Unsloth makes good quants:

https://docs.unsloth.ai/models/tutorials-how-to-fine-tune-an...

Note that llama.cpp doesn't try to be production-grade engine, more focused on local usage.