[radarsat1]

The Turing Machine idea makes a lot of sense... the machine is simply a state machine graph that interacts with the memory --- sensible that it could be "learned" similar to any genetic algorithm approach. Pretty cool trick regarding the differentiability of the system however.

That said, the biggest challenge here, I imagine, is evaluating the learned system. It may give right answers, but how often does it give wrong answers? How can the learned "machine" be tested for correctness? How does overfitting come into the picture? For instance, halting cannot be proved nor guaranteed. This strikes me as a fundamental advantage of a more functional "feed forward" approach of most learning systems.

[shawntan]

The paper discusses putting the NTM through several tasks, and tests for "overfitting" or how well it has generalised the task by giving it a slightly longer task than it has seen during training. For example, in the copy task, they trained it on sequences of length 20, but tested the it on a sequence of length 100.

Of course, this doesn't guarantee anything, but they also take a look at some of the internals of the learnt system which are more easily interpreted, and found that it does some pretty consistent things.

[noiv]

In many cases there are no correct answers only adequate ones. Did I chose the right job, car or partner? Who can tell?