[sjg007]

NNs can approximate any computable function. Any function can also be approximated by a polynomial. This is all proven math and has been known for a long time. I think of these as different facets of the same thing and each has different tools for computing, analyzing and understanding.

[lars]

A hash table can also approximate any function, so the "universal function approximation" thing is a bit oversold. It isn't really what matters. What matters is how well methods generalize beyond the training data, and how much data they need to do this well.