[stopachka]

I don't quite understand, what would 100K buy you?

AFAIK you would get about ~5 concurrent users, with a max context window of ~128K tokens on the larger models.

This wouldn't be good enough for coding -- are you guys thinking of using it for something else?

[bendews]

By my calculations 100k could get you 18 5090's + compute to host them, or 18 96gb Mac mini's. You can get a lot of context window and users out of that setup.

[FuriouslyAdrift]

Gigabyte 4x AMD Instinct MI300A rack server (512GB GPU RAM total)

Roughly equivalent to 4x H200's for less than half the price.

Vaguely around 60k tokens per second...