[nurettin]

I'm running it on a pure cpu 2020 model ryzen server with 2x128 GB RAM with llama.cpp, it seems as intelligent as gpt4. I optimized a little bit by forcing it to run on a single RAM stick and tuning llama.cpp build parameters, going from 3-5 tok/s to a more acceptable 5-8 tok/s.

It can call tools and reason adequately enough to use them when appropriate.

[ramgine]

Does keeping it on one stick make it more performant? I have a epyc server with 1tb of 64 gig sticks and a 3060, looking to maximize what models I can run on it

[nurettin]

In my case it definitely helps, but keeping 64k context window would probably require more than 64 GB.