[godelski]

I'm not sure if the Bitter Lesson is wrong, I think we'd need clarification from Sutton (does someone have this?)

But I do know "Scale is All You Need" is wrong. And VERY wrong.

Scaling has done a lot. Without a doubt it is very useful. But this is a drastic oversimplification of all the work that has happened over the last 10-20 years. ConvNext and "ResNets Strike Back" didn't take off for reasons, despite being very impressive. There's been a lot of algorithmic changes, a lot of changes to training procedures, a lot of changes to how we collect data[0], and more.

We have to be very honest, you can't just buy your way to AGI. There's still innovation that needs be done. This is great for anyone still looking to get into the space. The game isn't close to being over. I'd argue that this is great for investors too, as there are a lot of techniques looking to try themselves at scale. Your unicorns are going to be over here. A dark horse isn't a horse that just looks like every other horse. Might be a "safer" bet, but that's like betting on amateur jockies and horses that just train similar to professional ones. They have to do a lot of catch-up, even if the results are fairly certain. At that point you're not investing in the tech, you're investing in the person or the market strategy.

[0] Okay, I'll buy this one as scale if we really want to argue that these changes are about scaling data effectively but we also look at smaller datasets differently because of these lessons.