[pianopatrick]

To run a local AI that is half decent at research at usable speeds requires hardware that costs thousands of dollars.

Spending thousands of dollars on hardware to save dollars per month on tokens does not make financial sense.

If you run the numbers you'll probably find that using cheaper cloud models makes more financial sense than running those same models locally.

[Bender]

I suppose I was thinking in terms of companies that already had a slew of hardware and are just missing external GPU's, FPGA's or whatever people are using these days. A company might have a $50MM budget for tokens but could save a chunk of that using their existing hardware with some modifications.

[pianopatrick]

Maybe that makes sense if the company already bought the hardware.

But I currently think the fundamental reason the cloud models often make more sense is based in how the models work. My current understanding is that once these models are loaded into GPU memory, the model can respond to multiple prompts in parallel. So a cloud vendor that receives lots of user prompts at the same time can achieve really good hardware utilization. These cloud vendors can then spread the cost of the hardware over many more users than you can with a local model, leading to lower prices per prompt in the cloud.

[graemep]

If you are an individual, almost certainly, unless you are using LLMs very heavily.

If you are a business employing many people all of who use LLMs reasonably heavily? You can still spread the hardware cost over many users and respond to multiple prompts in parallel. At some point buying the hardware will make sense.

I would not get a dedicated server to run a single small VPS. On the other hand if I needed a lot of capacity a dedi is cheaper than a big VPS (or lots of small ones).

There is also a benefit for anyone who has confidential or secret information.