Hacker News new | ask | show | jobs
by nuancebydefault 1182 days ago
AI chip making? I can train an AI on my intel laptop if I whish. If I need more CPU power, i can rent some. The genie is out of the bottle and the only way is forward. The latest worldwide race.
1 comments

This isn't accurate. The bottleneck in very-large-scale-training BY FAR is communication between devices. If you have a million CPUs, the communication cost will be significantly higher than a thousand A100s (perhaps in the order of 100x or even more). So this is only possible to replicate with very dense and high compute chips with extremely fast interconnect.
Thanks for providing this insight. Is A100 the only platform? Can we pause/resume all such platforms simultaneously?