[ninkendo]

If LLM’s were local and cheap, sure. They’re just too heavyweight of a tool to use for simple CLI output manipulation today. I don’t want to send everything to the cloud (and pay a fee), and even if it was a local LLM, I don’t want it to eat all my RAM and battery to do simple text manipulation.

In 20 years, assuming some semblance of moore’s law still holds for storage/RAM/gpu, I’m right there with you.

[d3nj4l]

On my M1 Pro/16GB RAM mac I get decently fast, fully local LLMs which are good enough to do this sort of thing. I use them in scripts all the time. Granted, I haven’t checked the impact on the battery life I get, but I definitely haven’t noticed any differences in my regular use.

[mosselman]

Which models do you run and how?

[verdverm]

https://github.com/ggerganov/llama.cpp is a popular local first approach. LLaMa is a good place to start, though I typically use a model from Vertex AI via API

[mosselman]

Thanks. I have llama.cpp locally. How do you use it in scripts? As in how do you specifically, not how would one.

[d3nj4l]

I have ollama's server running, and I interact with it via the REST API. My preferred model right now is Intel's neural chat, but I'm going to experiment with a few more over the holidays.

[verdverm]

I tried ollama today and it is super easy, finding good models is definitely going to be challenging. I tried a few on some sample (JSON) tasks and it is... frustrating... how they ellide or are unable to follow instructions.

Is there a good fine-tuning workflow with ollama?

[d3nj4l]

I use ollama (https://ollama.ai/) which supports most of the big new local models you might've heard of: llama2, mistral vicuna etc. Since I have 16GB of RAM, I stick to the 7b models.

[_joel]

not op, but this is handy https://lmstudio.ai/

[mosselman]

Thanks!

[chongli]

Yeah, it would be much better if you could send a sample of the input and desired output and have the LLM write a highly optimized shell script for you, which you could then run locally on your multi-gigabyte log files or whatever.