[Wiretrip]

In the text generation tool outlined above (and indeed many of the convnet-based visual networks), the hidden layers are there precisely to extract 'meaning'. The lower layers (closer to the source input) deal with syntax and feed upwards to hidden layers that extract semantic features, which in turn feed upwards to more layers, each with a bigger overview of the semantic features and thus ultimately the context. That's the idea anyway.

[foldr]

>the hidden layers are there precisely to extract 'meaning

That is just wishful thinking, no? I mean, there is no particular reason to think that the hidden layers will actually do this with any high degree of success.