I'm running it on a pure cpu 2020 model ryzen server with 2x128 GB RAM with llama.cpp, it seems as intelligent as gpt4. I optimized a little bit by forcing it to run on a single RAM stick and tuning llama.cpp build parameters, going from 3-5 tok/s to a more acceptable 5-8 tok/s.
It can call tools and reason adequately enough to use them when appropriate.
Does keeping it on one stick make it more performant? I have a epyc server with 1tb of 64 gig sticks and a 3060, looking to maximize what models I can run on it
It can call tools and reason adequately enough to use them when appropriate.