[simiones]

> They can theoretically model any function, but the number of parameters needed means in practice they can't.

Even theoretically, no they can't. They can theoretically model any continuos function.

Plus, even for continuous functions, the theorem only proves that, for any function, there exists some NN that approximates it to arbitrary precision. It is not known whether there is some base NN + finite training set that could be used to arrive at that target NN using some algorithm in a finite number of steps.

[nightski]

I'm not sure it is all that interesting of a distinction seeing as non-continuous functions can be approximated by continuous ones (basically the entire premise of a digital computer).

[simiones]

I don't think this is right at all. Digital computers express non-continuous functions, and they sometimes use those to approximate continuous functions.

For example, for a function f(x) defined on R with f(x) = -x if x < 0 and f(x) = 7+x if x >= 0, how would you approximate it by a continuous function g(x) with precision lower than, say, 1 (i.e. |f(x) - g(x)| < 1 for any x in R)?

And of course, there are functions with much worse discontinuities than this.

[IshKebab]

He was talking about the actual signals in chips being continuous I believe.

[GranularRecipe]

This is not only an issue for shallow networks. As far I know, both points apply to all feed-forward networks regardless of depth.

[simiones]

Yes, both apply regardless of depth (as long as it is finite, I imagine).