[fwip]

What? LLMs are perhaps the least bandwidth-constrained thing you can do over the network.

If you run a local model at a good pace, you're emitting tokens at, what, 60 tokens/sec? That's, very roughly, 300 bytes/sec. You can get that speed with dialup from the 80's! And when it takes a half second just to ingest the input tokens (to get to the part where you start generating the reply), you're not gonna care about even a crazy 250ms latency.

Unless you think that the model should be downloaded on-the-fly in response to a user request, there's zero reason for an LLM to need any sort of fast connection.