[Zemtomo]

This is bleeding edge stuff.

All advances in this direction do indicate that it will be easier and easier for more people to do things with it.

This doesn't need to work for everyone.

A 4090 costs today 2k, the 3090 with also 24gb costs today 1k and costed 2k.

[caeril]

Any particular reason not to run this model on a single Jetson AGX Orin 64GB?

GP Core count is much lower than than the 4090 but it still does 275 int8 TOPS for only $2k

[Zemtomo]

I'm the wrong person to ask but performance wise a 4090 has over a pflop and a Google search also showed a factor of 3-4 for a 3090.

[cjbprime]

The relevant performance metric here is memory amount and bandwidth.