[pobonin]

I must say, this is incredible, nice work! I've been tinkering with local LLMs for a while now and experimenting with different tech, and I'd never thought it'd be possible to run such a large model on consumer-grade compute. It still feels like these days, you need to buy a $10K DGX Spark or something crazy expensive just to get to that level of performance but with experiments like yours, it looks like there's still a lot of room for improvement and innovation.

What I'd like to see personally is Small Language Models giving similar performance eventually (because do we really need that many hundred billion parameters?). Researchers have shown that a lot of attention heads in the LLMs we're using these days are barely even active, so the MoE architecture seems like a step in the right direction. I still wonder how far we can push things back towards a 1B or 3B parameters model while trying to reach similar performances as double-digit or triple-digit B params models.