[zeta0134]

You are vastly overestimating the startup cost. For me this week it was literally these commands:

pacman -S ollama

ollama serve

ollama run llama3

My basic laptop with about 16 GB of RAM can run the model just fine. It's not fast, but it's reasonably usable for messing around with the tech. That's the "startup" cost. Everything else is a matter of pushing scale and performance, and yes that can be expensive, but a novice who doesn't know what they need yet doesn't have to spend tons of money to find out. Almost any PC with a reasonable amount of RAM gets the job done.

[monkmartinez]

llama3 at 8billion params is weak sauce for anything serious, it just isn't in the same galaxy as Sonnet 3.5 or GPT-4o. The smaller and faster models like Phi are even worse. Once you progress past asking trivial questions to a point where you need to trust the output a bit more, its not worth effort in time, money and/or sweat effort to run a local model to do it.

A novice isn't going to know what they need because they don't know what they don't know. Try asking a question to LLaMA 3 at 8 billion and the same question to LLaMA 3 at 70 billion. There is a night and day difference. Sonnet, Opus and GPT-4o run circles around LLaMA 3 70b. To run LLaMA at 70 billion you need serious horse power as well, likely thousands of dollars in hardware investment. I say it again... the calculus in time, money, and effort isn't favorable to running open models on your own hardware once you pass the novice stage.

I am not ungrateful that the LLaMA's are available for many different reasons, but there is no comparison between quality of output, time, money and effort. The API's are a bargain when you really break down what it takes to run a serious model.

[jononor]

Using an LLM as a general purpose knowledge base is only one particular application of an LLM. And on which is probably best served by ChatGPT etc.

A lot of other things are possible with LLMs using the context window and completion, thanks to their "zero shot" learning capabilities. Which is also what RAG builds upon.

[Aurornis]

I’m familiar with local models. They’re fine for chatting on unimportant things.

They do not compare to the giant models like Claude Sonnet and GPT4 when it comes to trying to use them for complex things.

I continue to use both local models and the commercial cloud offerings, but I think anyone who suggests that the small local models are on par with the big closed hosted models right now is wishful thinking.