[sanderjd]

I honestly think it's still way too early to say this either way. If your hypothesis that there are no breakthroughs left is right, then it's still a very big deal, but I'd agree with you that it's not steam engine level.

But I don't think "the transformer paper was eight years ago" is strong evidence for that argument at all. First of all, the incremental improvement and commercialization and scaling that has happened in that period of time is already incredibly fast. Faraday had most of the pieces in place for electricity in the 1830s and it took half a century to scale it, including periods where the state of the art began to stagnate before hitting a new breakthrough.

I see no reason to believe it's impossible that we'll see further step-change progressions in AI. Indeed, "Attention is All You Need" itself makes me think it's more likely than not. Out of the infinite space of things to try, they found a fairly simple tweak to apply to existing techniques, and it happened to work extremely well. Certainly a lot more of the solution space has been explored now, but there's still a huge space of things that haven't been tried yet.