[eli_gottlieb]

More like "the set of things you can do with functions, where you've got a strong prior on the functions being things like spacial mappings with certain invariances." So basically more like, "functions for vision problems."

[tMcGrath]

You are absolutely right that a lot of work to date (especially in applications and metrics-driven research) has focused on 'functions for vision problems'. But this is far from the only idea - that's the point I'm trying to make.

For instance what about learning value functions for reinforcement learning (e.g. AlphaGo)? Or natural language processing? These are definitely not vision problems, or if you believe that they are then 'functions for vision problems' is actually a pretty huge class!

The universal approximation theorem backs up my claim [0] - we can approximate arbitrary functions with neural networks. I think this theorem is overemphasised in practice: we don't generally want to approximate arbitrary functions, we _want_ to encode specific prior information into the function we approximate, as you rightly say. But that doesn't mean that we have to do so, or that we only have one idea about what functions to encode, or even how to encode them.

[0] https://en.wikipedia.org/wiki/Universal_approximation_theore...

[eli_gottlieb]

>The universal approximation theorem backs up my claim [0] - we can approximate arbitrary functions with neural networks. I think this theorem is overemphasised in practice: we don't generally want to approximate arbitrary functions, we _want_ to encode specific prior information into the function we approximate, as you rightly say. But that doesn't mean that we have to do so, or that we only have one idea about what functions to encode, or even how to encode them.

I wouldn't say we have only one idea about functions, but I would say I haven't seen much of an active pipeline, outside maybe DeepMind, on coming up with new kinds of priors over functions that we can apply to larger-scale or more structured tasks. At some point, applied deep learning will run out of steam, and someone's going to have to go back to doing basic research.

That someone may find, as many have, that in terms of sample efficiency and transfer learning, deep neural nets are not always so great.