[mempirate]

Thanks for the detailed response. My goal is to deploy small to medium-sized LLMs on it for things that I'd rather keep private and local, i.e. automation around smart home, but also tracking finances and portfolio, aggregating & summarizing news feeds and blog posts and so on. For these purposes, medium intelligence at low TPS seems good enough to me.

But a large part of it is also wanting to learn more about serving these models, and potentially some small-scale experiments with training and fine tuning.