[kcb]

What's the benefit to running LLMs locally? Data is already remote, LLM inferencing isn't particularly constrained by Internet latency. So you get worse models, performance, and battery life. Local compute on a power constrained mobile device is required for applications that require low latency or significant data throughput and LLM inferencing is neither.

[fkyoureadthedoc]

> What's the benefit to running LLMs locally?

At work:

That I don't rent $30,000 a month of PTUs from Microsoft. That I can put more restricted data classifications into it.

> LLM inferencing isn't particularly constrained by Internet latency

But user experience is

[kcb]

30k in a month is an enormous amount of tokens with Claude through AWS Bedrock. And companies already commonly trust AWS with their most sensitive data.

[mrweasel]

The data you need is mostly not remote. A friend works at a software development company, they can use LLMs, but only local ones (local as in their datacenter) and it can only be trained on their code base). Customer service LLMs need to be trained on in-house material, not generic Internet sources.

The general advantage is that you know that you're not leaking information, because there's nowhere to leak it to. You know the exact input, because you provided it. You also get the benefit of being able to have on device encryption, the data is no good in the datacenter if it's encrypted.

[kcb]

Local as in datacenter is the key there. The original comment was about end user devices.