[ttul]

You are not wrong, but the practical reality of local hardware is to be batch-constrained in comparison with a multi-user inference cloud. You will never be able to compete cost effectively in your home lab with a cloud that has >100M end users streaming millions of inference requests per second across a gigantic fleet of machines.

Can I run a few inferences in parallel on my Mac Mini? Yes. But put 1,000 Mac Minis in a datacenter serving 1,000 copies of myself? That's going to be more efficient.

[anon373839]

Oh certainly, a local machine can't parallelize like the cloud can. Does anyone think that, though? What it can do is provide good-enough inference to support common business needs, without creating unnecessary data risks.

[dickersnoodle]

Why on earth would you need to do that for most things? Not everyone wants to be the next Coreweave. It sounds like you're killing a fly with a cannon to me.