[p1esk]

Pretend you have any hardware you want, today. What would you do with it? What model would you train? How do you know available hardware is the bottleneck and not model architecture?

[Legend2440]

Because with infinite hardware I'd be able to do neural architecture search and find the optimal model architecture.

And I'd be able to train a learned optimizer to replace gradient descent as the training process.

Even without either of those, performance improves in a predictable way with more compute thanks to scaling laws.