[time_to_smile]

> This is how neural networks work: they see many examples and get rewarded or punished based on whether their guesses are correct.

This description more closely describes reinforcement learning, rather than gradient based optimization.

In fact, the entire metaphor of a confused individual being slapped or rewarded without understanding what's going on doesn't really make sense when considering gradient optimization because the gradient wrt the to loss function tells the network exactly how to change it's behavior to improve it's performance.

This last point is incredibly important to understand correctly since it contains one of the biggest assumptions about network behavior: that the optimal solution, or at least good enough for our concerns solution, can be found by slowing taking small steps in the right direction.

Neural networks are great at refining their beliefs but have a difficult time radically changing them. A better analogy might be trying to very slowly convince your uncle that climate change is real, and not a liberal conspiracy.

edit: it also does a poor job of explaining layers, which reads much more similar to how ensemble methods work (lots of little classifiers voting) than how deep networks work.

[cshimmin]

Well said re: gradient optimization vs. "getting slapped". However, note that since NN optimization is almost always nonconvex, we are NOT guaranteed to arrive an a optimal (or even close-enough) solution. A major limitation of gradient based optimization on nonconvex problems is that they are very susceptible to getting trapped in local minima.

But, for now it's the best tool we have, so we just have to hope that we get close enough, or just empirically run lots of times to find the best local minimum we can. Incidentally, this actually is more like a brute-force approach, but at the ensemble level, which is quite different than the article means it.