[pizzao]

The required compute seems a bit high: "We trained all CIFAR-10 models on 1xB200 GPU, and all ImageNet 64×64 models on 8xB200 GPUs. The largest CIFAR-10 model uses 20 B200 hours to train, and the largest ImageNet 64×64 model uses 640 B200 hours"

20 B200 hours for CIFAR-10 seems like a lot...

[orbital-decay]

They're proposing a loose range of hardware that is thought to be more efficient when used with this thing, that's the entire point. What they did with B200s is just a simulation, it's not supposed to be GPU-efficient.