Probably I did this wrong but I’m getting an approximation of 300K H100s completes that in a month. At least they choose something fairly large it seems. Not sure how LoRA or other incremental training is handled.
Depends on which spec you used, since the law doesn't specify the floating point width. If you used FP8 ops on the H100 SXM then a single GPU would hit the limit in 25265285497.72612 seconds. 300,000 GPUs would pass 10^26 FP8 ops in 23 hours.