[LoganDark]

They need 568 LPUs to load both Mixtral 8x7B and LLaMA 70B, because they need both those models available for the demo.

I imagine Mixtral by itself would only take something like 200-300 LPUs

[wmf]

Only $5M then.

[LoganDark]

I'm pretty sure $20,000 per LPU isn't actually the cost of these LPUs. I saw someone else on HN asking if $20,000 could get them something and an employee said to reach out. Which makes me think $20,000 is enough to get some sort of model running at least, even if it's not necessarily an LLM.

[int_19h]

$5M once, upfront. But given the significantly increased throughput, how fast does that pay for itself?

[fzzzy]

You need computers for all of them and megawatts of power, power supplies, cooling, and power distribution.

[int_19h]

Naturally, but you need that for GPUs as well, no? What is the actual difference when running, when measured per token generated?

[gessha]

Depends on power usage. I’m curious how power hungry those are compared to server/workstation cards.