Can someone go in to more detail about how certain neural networks inherently are able to encode and interpret particular structures? Why RNs capture relations and Recurent neural networks can capture sequential data?
Basically, you can encode your (human) assumptions/intuitions about the data/task into the math. Neural networks are just big math equations with a bunch of tunable parameters, and there is a lot (a LOT) of freedom in what those math equations can be. So, CNNs assume spatial locality (stuff being near each other) matters, which turns out to be super true for images - an image is not at all the same if you just shuffle all of its pixels around. All it takes to do that is a simple tweak to the math (make a small set of weights that is the same for a bunch of patches of the input, instead of separate weight per input). This RN concept is also a simple math tweak, it's just a function designed to do a pairwise sum over pairs of 'objects', which they claim is a good way to enable the neural network to learn about relational concepts. RNNs are also a simple math tweak, just make the function recursive (sort of, it's trickier in implementation, but close enough).