[f1shy]

Wait! We certainly did NOT have huge datasets (like current internet) for ages. Not even decades. I’ve seen a lecture by a MIT professor (which I cannot find now) where he asserted categorically, that the advances in AI are mostly because of the huge data that we now have and we didn’t before. And that was an old video.

[yosefk]

Whichever way it's true in, it's not true in the sense that eg you can approximate any curve with a single layer neural net, and you're not actually going to be able to do it for problems CNNs or transformers work decently on. And Google indexed all of the public Internet way before its researchers came up with transformers.

Another way to look at it is that like you say, it was an old video but there has been progress since though we had large datasets when it came out by its own definition