[trentonstrong]

It's interesting since Hinton's approach seems to be more inspired by the computational geometry aspects of vision (reverse computer graphics, invariant representations) and working backwards towards the neural superstructures.

Hawkins' seems to be inspired by the biology itself, working from the "common computational substrate" hypothesis up to the cortical units required.

They both seem to meet in the middle at the need to figure out invariant representations for the intermediate features presented to each level of the hierarchy.

I do wonder what an argument between those two would look like concerning the applicability of back-propagation, which I remember Hawkins' deriding as totally artificial compared to the feedback structure of the actual neocortex.

Anyone more up to date on the state of the argument regarding that?

[MrQuincle]

Backpropagation is just calculating the gradient of a multilayer perceptron. You do not even have to calculate it, you can use autodiff for it: http://users.cecs.anu.edu.au/~jdomke/courses/sml2010/07autod... (pdf).

They both seem to neglect time (internal dynamics). I haven't seen either of them coming up with a model as that from Izhikevich with polychronization: http://www.izhikevich.org/publications/spnet.htm. If we would be able to make one of such ideas computationally useful, things would become really interesting.

[trentonstrong]

Interesting, thanks for the links!

I've done your typical ANN 101 training in the past so have a good mental model for back-propagation. Modelling the actual nonlinear dynamics of "realistic" neural networks seems like an obvious path of research but I know how daunting it is. It seems like every tiny bit we can push forward our tools for understanding complex non-linear systems should pay large dividends across so many different computational fields (fluid dynamics, QFT, economics, ..., everything?)

I'll have to read Izhikevich's paper, seems like a unique line of research.

[MrQuincle]

Absolutely nice to check his work. He founded a startup, Brain Corporation (http://www.braincorporation.com/) recently.

Coincidence detection where delays are playing a functional role is one of the things that I find interesting (as well as the fact that there are more polychronous groups than neurons). The other thing is the emergence of gamma waves. I would be surprised if these do also not have some functional role (although it might be just as well the humming of our biological processor). :-)

I wish I was brave enough to start experimenting with different neural networks. For now I am on the roll "Bayesianfying" everything I encounter. Even the Hough transform that Hinton is so fond of in this talk. :-)

[jerf]

If I'm following this presentation correct it seems like it would be relatively simple to add time into these models by using the previous frame's guesses as priors for this frame.

He offhandedly mentioned something once that may have been this, but there wasn't enough context to be sure this is what he meant. Still, with the general idea of this reliable high-level affine invariance it doesn't seem hard to imagine how to convert this to a temporally-aware approach, at least at a basic level.

[MrQuincle]

At the end of his talk he answers a question about audio. A student of him has worked on it and he expressed admiration for that student. He didn't come across as if he solved that problem as well. :-)

Time is a tricky bastard. It is one thing to incorporate some Markovian dynamics in an artificial network, it is another thing to cope with dynamics as we know it. You might be interested in for example something like what Ralf Der is studying: http://www.informatik.uni-leipzig.de/~der/ (Tishby is working on this as well: https://www.cs.purdue.edu/homes/spa/venice08/docs/Venice-Tih... (pdf, slides)).

Two things I learned from his talk:

* If you have (human-like) part-based representations you need linear relationships to build up the whole again.

* Routing is key.

With dynamics involved, say movie scenes, or your hand moving in front of your own eyes, we might postulate that we will first discover linear relationships as well. A car disappearing temporarily behind a fence on the right we will expect to appear on the other side at the left.

Routing in the brain that corresponds with dynamic behavior, is probably something else than the time slicing / windowing we are all so familiar with in machine learning. It also goes beyond a simple central pattern generator for locomotion, which is not a routing problem at all (but just a clone of the outside frequency pattern with dynamics between neurons). "Real" dynamics is about being able to put items into slots, to learn a (visual) grammar. The thought that routing is key seems also be shared by Schmidthuber who created LTSM only to be able to route errors in a more sophisticated way through a network.

[cowpig]

The problem is that Hawkins hasn't really found a comparable alternative. He's trying to throw away decades of progress in mathematics and reinvent the learning algorithms, which is great, but he hasn't succeeded (yet)