[MuffinFlavored]

Because I pay $20/mo for GPT-4 and don't understand why anybody would run a "less-good" version locally that you can trust less/that has less functionality.

That's why I wanted to try to understand, what am I missing about local-toy LLMs. How are they not just noise/nonsense generators?

[jstarfish]

Sometimes you just need to write creative nonsense. Emails, comments, stories, etc. Great for fiction since there are low stakes for errors.

They're bad at generative tasks. Don't have it write code or scientific papers from scratch, but you can have it review anything you've written. You can also do summaries, keyword/entity extraction, and the like safely. Any reductive task works pretty well.

[tvararu]

So do I, but on a flight 2 days ago, I forgot the name of a Ruby method, but knew what it does. I tried looking it up in Dash (offline rdocs) but didn’t find it.

On a whim, I asked Zephyr 7B (Mistral based) “what’s the name of that Ruby method that does <insert code>” and it gave me 3 different correct ways of doing what I wanted, including the one I couldn’t remember. That was a real “oh wow” moment.

So offline situations is the most likely use case for me.

[ParetoOptimal]

> don't understand why anybody would run a "less-good" version locally

Privacy.

If you use llm's on private documents you don't want others to see, you'll likely prefer local models strongly.

[frankacter]

How would you, in LM Studio or another local option, populate with your own documents? I don't see any option of applying knowledge via pdf/doc/txt.

[ParetoOptimal]

https://github.com/imartinez/privateGPT

https://khoj.dev/

[RecycledEle]

> I pay $20/mo for GPT-4 and don't understand why anybody would run a "less-good" version locally that you can trust less/that has less functionality.

Local might still be there after an online service is no longer available.

[cellis]

The first reason is because GPT-4 is censored for many types of tasks/topics.