[apu]

I'd say both are somewhat true.

A lot of early "progress" in AI was found to not survive contact with the real world -- for example, most of computer vision. This was because collecting data was so expensive/difficult that only a few images could be captured for many experiments, and the methods they came up with often worked okay for those examples, but nothing else! So a lot of clever-seeming algorithms end up being rather useless in the real world, and progress was illusionary.

I find that in computer vision (my area of research), a fundamental component of many disparate problems is that you are trying to interpolate or extrapolate data in a very complicated underlying space, where linear approximations are completely unusable and optimization is too unconstrained. The key is to come up with suitable regularizers that can use prior information to constrain the problem appropriately.

Getting more data thus helps in two ways:

1. It reduces the amount of interpolation you have to do, since you can get a denser sampling of the space.

2. It allows you up to build up these priors using real data, making interpolation much better.