[utopiah]

That's nice... in theory. Like it could be cool, and useful... but like what would I actually run on it if I'm not a spammer?

Edit : reminds me of federated learning and FlowerLLM (training only AFAIR, not inference), like... yes, nice, I ALWAYS applaud any way to disentangle from proprieaty software and wall gardens... but like what for? What actual usage?

[utopiah]

Gimme an actual example instead of downvoting, help me learn.

Edit on that too : makes me think of OpenAI Whisper as a service via /e/OS and supposedly anonymous proxying (by mixing), namely running STT remotely. That would be an actual potential usage... but IMHO that's low end enough to be run locally. So I'm still looking for an application here.

[wat10000]

Are you looking for a general application of LLMs too large to run locally? Because anything you might use remote inference for, you might want to use privately.

[utopiah]

Sure that'd do, what

- useful thing (according to someone specific requirements, maybe hallucinations are OK, maybe not) that

- needs privacy (for example generating code that will be open source probably does not need that)

- can't be run locally

- can be trusted to actually process as said it does

[wat10000]

I've found LLMs to be extremely useful for writing helper scripts (debugger enhancements, analyzing large disassembly dumps, that sort of thing) and as a next-level source code search. Take a large code base and you get an error in component A involving a type in component B and it's not immediately obvious how the two are connected. I've had great success in giving an LLM the error message and access to the code and asking it how B got to A and why that's an error. This is something I could certainly do myself, but there are times when it would take 100x longer.

The key is that these are all things I can verify without much difficulty: read over the script, spot-check the analysis, look at the claimed connection between A and B and see if it's real. And I don't really care about style, quality, maintainability.

You certainly can run this locally, but anything that will fit into reasonable local hardware won't be as good.

I don't need to trust it to process as it says it does, because I'm verifying the output.

And as far as I'm concerned, "needs privacy" is always true. I don't care if the code will be open source. I don't care if it's analyzing existing code that's already open source. Other people have no business seeing what I'm doing unless I explicitly allow it. In any case, I work on a lot of proprietary code as well, and my employer would be most displeased if I were exposing it to others.

[fragmede]

> would I actually run on it if I'm not a spammer?

> Gimme an actual example instead of downvoting, help me learn.

Basically you asked a bunch of people on a privacy minded forum, why should they be allowed to encrypt their data? What are you (they) hiding!? Are you a spammer???

Apple is beloved for their stance on privacy, and you basically called everyone who thinks that's more than marketing, a spammer. And before you start arguing no you didn't, it doesn't matter that you didn't, what matters is that that's how your comment made people feel. You can say they're the stupid ones because that's not what you wrote, but if you're genuinely asking for feedback about the downvotes, there you are.

You seriously can't imagine any reason to want to use an LLM privately other than to use it to write spam bots and to spam people? At the very least expand your scope past spamming to, like, also using it to write ransomware.

The proprietary models that can't be run locally are SOTA and local models, even if they can come close, simply aren't what people want.

[utopiah]

Seems I formulated my question in a way that wasn't clear.

I specifically like the privacy aspect (even though honestly I think most people in this forum claim they do and yet they rely on BigTech with which they their data, so IMHO most people on HN are not as demanding as you describe) the question is precisely about what to run.

FWIW I specifically keep a page on self hosting AI (you can check if you want to see if it's real at https://fabien.benetou.fr/Content/SelfHostingArtificialIntel... ) so again, the privacy aspect is crucial to me.

The question is ... what to actually run. What can't be run locally that would be useful. What model for which tasks. For example coding (which I don't think models are good enough for) typically would NOT need this because one wouldn't share any PII over there, hopefully would even instead publish the resulting code as open source.

So my provocation about spammer is... because they often ARE actual users of LLMs. They use LLM for their language capabilities, namely craft a message that is always slightly different yet convey (roughly) the same meaning (the scam) to avoid detection. A random person though using LLM might NOT be OK with hallucinations when they use it for their own private journal or chat.

So... what for?

Edit: I did use Apple for years, recommending by someone at Mozilla, and I moved away from them earlier this year precisely because even though they are better than others, e.g Google, IMHO it's just not good enough for me. No intermediary is better than one with closed source.

[fragmede]

Okay, so you had a whole agenda. Could have just been more transparent if you came out and said that directly.

It's an LLM. The user can ask it anything they can think of! But then to to "only spammers could eke out anything from them".

What do users ask OpenAI about? My boyfriend/girlfriend/wife/mistress cheated on me, what should I do? My mom is... My dad is... My friends are.. I have this problem with this company and I want to sue them... I have this weird lump on my foot... More nefariously, I'm sure someone out there asking "how do I make cocaine" is seriously considering it, and not just testing the machine. I want to talk to somebody about 1994, the TV series from 2019. I want to write a fiction book about the near future but one where I won the lottery or I grew up rich or I was Harry Potter or a murder mystery or utopian sci-fi or dystopian sci-fi or an alt-history where there are still dinousaurs or or or.

I don't know if it's a failure of your imagination, or if mine is overactive, but making a venn diagram of all the world's humans broken down into spammers and not spammers, and then placing the circle users of local LLMs inside of spammers, and there's no one else, just seems a bit reductive.