Yep, hetero multigpu fleet mixing high ram GPUs (40-80GB each on each A100) as multigpus w smaller (ex: ~12-16 GB T4s) nodes, w crazy interconnects locally (nvlink) and across nodes. And storage gets fun as well, like parallel SSD arrays for 100GB+/s combined per node. Then whatever legacy+hybrid CPU stuff. Ex: for stuff like PCIe, new generations that ~10x the bandwidth you'd see in a gamer box, and like 1-2 per GPU. Varies a lot for say log mining vs NN training, and even for diff NNs. Ex: Graph NNs end up needing more balanced CPU side.
Saturating a box with 500+ GB GPU RAM is fun. Only our gov users ask us for help on that typically: most of our users are commercial nowadays, but with much smaller/scaled down GPU rigs. I think that'll change as the fintechs keep improving and software gets easier, but they are still not there (outside of niches). Working on it :)
Saturating a box with 500+ GB GPU RAM is fun. Only our gov users ask us for help on that typically: most of our users are commercial nowadays, but with much smaller/scaled down GPU rigs. I think that'll change as the fintechs keep improving and software gets easier, but they are still not there (outside of niches). Working on it :)
(If you like writing shaders, we are hiring :D )