[d3nj4l]

On my M1 Pro/16GB RAM mac I get decently fast, fully local LLMs which are good enough to do this sort of thing. I use them in scripts all the time. Granted, I haven’t checked the impact on the battery life I get, but I definitely haven’t noticed any differences in my regular use.

[mosselman]

Which models do you run and how?

[verdverm]

https://github.com/ggerganov/llama.cpp is a popular local first approach. LLaMa is a good place to start, though I typically use a model from Vertex AI via API

[mosselman]

Thanks. I have llama.cpp locally. How do you use it in scripts? As in how do you specifically, not how would one.

[d3nj4l]

I have ollama's server running, and I interact with it via the REST API. My preferred model right now is Intel's neural chat, but I'm going to experiment with a few more over the holidays.

[verdverm]

I tried ollama today and it is super easy, finding good models is definitely going to be challenging. I tried a few on some sample (JSON) tasks and it is... frustrating... how they ellide or are unable to follow instructions.

Is there a good fine-tuning workflow with ollama?

[d3nj4l]

I haven’t tried any fine tuning so I can’t help there, sorry. Though I will say that neural chat has been pretty good to me, even though I have definitely observed it ignoring instructions at times, like a “here’s your json:” preamble in response to a query that specifically requests only json.

[d3nj4l]

I use ollama (https://ollama.ai/) which supports most of the big new local models you might've heard of: llama2, mistral vicuna etc. Since I have 16GB of RAM, I stick to the 7b models.

[_joel]

not op, but this is handy https://lmstudio.ai/

[mosselman]

Thanks!