Hacker News new | ask | show | jobs
by Alpha3031 8 days ago
Well, if you're happy with around (as in within an order of magnitude or two of) 0.1 tokens per second... I believe that's around what people are getting when loading MoE weights from NVMe.
1 comments

You've got to consider how much power that uses though. Depending where you live, some of these providers can serve it for less than you pay for power.