[EmbarrassedFuel]

A small bone of contention: it’s not about Moore’s law per se, which has been ‘dead’ since about 2013, coincidentally when the deep learning revival started. It’s matrix multiplication ASIC development that is driving the progress.

GPUs already existed when the idea to use them to make the feasible size of neural nets larger came about. For a long time the drive for the increase of GPU compute power was still gaming/commercial graphics houses. It’s really only in the last 1-2 years that we’ve seen highly specialised GPUs with features like tensor cores (or indeed google’s TPUs).

Also, calling neural nets ‘brute force’ because they use a lot of computing power to train a model is slightly reductionist - a true brute force approach to image recognition, ie enumerating all possible combinations of, say, 200x200x256x3 pixels, would be completely absurd and probably exceed the computing power currently available on earth.

[quanticle]

I'm not saying that neural nets are brute force. I'm saying that there haven't been any algorithmic improvements in neural nets to make them more computationally feasible than they were when they were first invented. Instead, we have specialized hardware which can just do the necessary computation quickly enough to make neural nets feasible.

It's not like neural nets are a new technology. They've been known since the '80s, at least. It's just that they were considered a dead end, because we didn't have the computational resources to run deep neural nets, nor did we have sufficient training data to make neural-net approaches feasible. Once those preconditions were met, neural nets took off in short order.

[EmbarrassedFuel]

There have - convolutional nets have far fewer weights to learn than an equivalent fully connected net of the same depth.

And you literally described it as a brute force approach in your comment.