Hacker News new | ask | show | jobs
by embedding-shape 2 days ago
> The Kimi-K2.6 model is 1.1T parameters with 32B active parameters. With light quantization (Q6_K) that's enough to run it (slowly) on a single 5090

Without leveraging system RAM and/or SSDs, I don't think you can, or how exactly are you running this, if this is something you are doing today? With CPU/expert offloading you could probably do it with a 5090 + 1TB of RAM or something like that, but absolutely not on a single 5090 entirely within VRAM.
2 comments
robotswantdata 2 days ago
Yes hybrid approaches are much better than people realise.

There are a lot of optimisations that are not in the public sphere, source working on start up in this space

link
embedding-shape 2 days ago
> There are a lot of optimisations that are not in the public sphere

Sure, but if we're participating in public discussions, isn't it more fun if we talk about things people can actually read and understand, rather than secret stuff other's can say work, but no can actually validate or know how it works?

It sounds like "hybrid approaches are much better than the public is aware, because everything else is private and secret", but also: ok, so what? No one can run that anyways, (yet?), so why it matters?

link
rhdunn 2 days ago
Yes, that's what I was saying w.r.t. expert offloading, i.e. ensuring that the GPU could fit the active parameters not all the parameters.
link
embedding-shape 2 days ago
Alright, I guess I misunderstood. To be fair, this part:

> The Kimi-K2.6 model is 1.1T parameters with 32B active parameters. With light quantization (Q6_K) that's enough to run it (slowly) on a single 5090.

Is painting a very different perspective, even considering the latter parts it's hard to read that as "Of course offloading everything else that doesn't fit on the GPU itself". But anyways, it's been clarified now so no harm :)

link