[eli_gottlieb]

>The universal approximation theorem backs up my claim [0] - we can approximate arbitrary functions with neural networks. I think this theorem is overemphasised in practice: we don't generally want to approximate arbitrary functions, we _want_ to encode specific prior information into the function we approximate, as you rightly say. But that doesn't mean that we have to do so, or that we only have one idea about what functions to encode, or even how to encode them.

I wouldn't say we have only one idea about functions, but I would say I haven't seen much of an active pipeline, outside maybe DeepMind, on coming up with new kinds of priors over functions that we can apply to larger-scale or more structured tasks. At some point, applied deep learning will run out of steam, and someone's going to have to go back to doing basic research.

That someone may find, as many have, that in terms of sample efficiency and transfer learning, deep neural nets are not always so great.