[sudohackthenews]

People have gotten manageable results on all sorts of hardware. People have even squeezed a few tokens/second out of Raspberry PIs. The small models are pretty performant- they get good results on consumer gaming hardware. My 2021 laptop with a 3070m (only 8gb vram) runs 8b models faster than I can read, and even the original M1 chips can run the models fine.

[monkmartinez]

You are right of course.... IF your metric for manageable/useable is measured only tokens per second (tok/s).

If your metric is quality of output, time, money and tok/s, there is no comparison; Local models just aren't there yet.