[shepherdjerred]

Weights are just numbers (probably floats?), right?

This model has 3.12TB of floats??? That's insane. How do you load that into memory for inferencing?

[exikyut]

Use x1e.32xlarge on AWS with 3TB of RAM. Just $12,742/mo - https://calculator.aws/#/estimate?id=7428fa81192c57087ac8cdf...

Alternatively order something like the HP Z8 with 3TB RAM configured, which is only $75k - https://zworkstations.com/configurations/2040422/

It's interesting. It would take ~six years for the Z8 to break even compared to AWS, but traffic into and out of the machine would be $0, and I don't think you're running directly on the metal with AWS, so performance would probably be a bit higher. And then there's storage - I configured, uhh, 120TB of a mixture of SSDs and HDDs. I'm not even going to try and ask AWS for a comparible quote there.

I may or may not have added dual Xeon Platinum 8280s to the Z8 as well. :P

[tjbiddle]

When you're spending that kind of money on a machine, there's no way you're paying retail price. Sales reps would give you a significant discount.

Also - think you meant 6 months, not 6 years anyhow :)

[exikyut]

Interesting. I'm very curious... 20%? 35%?

And I did mean 6 months, woops. Didn't even notice...

[jsnell]

> It would take ~six years for the Z8 to break even

Do you mean six months?

[exikyut]

Oh *dear*. I definitely tripped over there, and I didn't even notice.

Yup.

[high_byte]

Z8 sounds like fun. but I might just buy two teslas (roadster and X, or a cybertruck) and a gaming PC. :D

[visarga]

hate to break the party but this model only loads a small part of itself in RAM when inferencing

[exikyut]

That's a good thing. Less completely means more energy for interestingness, and less expense means more accessibility.