[SuchAnonMuchWow]

You would need ~250 groq cards to run a 7B model since their system doesn't scale down. So if you want to buy their hardware, you need a few millions dollars.

Their hardware was never for people at home, but for cloud providers.

[LoganDark]

That doesn't sound right. Their public demo ran on 568 LPUs because they had Mixtral-8x7B and LLaMA-70B (45B and 70B respectively). IIRC their cards each have slightly over 200MB of SRAM so this almost exactly checks out.

A 7B model would then be able to run on about 60 LPUs. Even at $20,000 per card that would be only $1.2 million and I highly doubt the cost is actually that high, that's just what DigiKey says the cost of an LPU is, if you're trying to buy just one :)