[zo7]

I don't think I or the parent comment are necessarily suggesting that more layers are better, but are pointing out that the fact that they're only using 5 layers suggests that they're not using a state of the art architecture. You can't faithfully say "oh a CNN cannot model this relationship" when it wasn't a thorough evaluation. (especially given that they don't mention modern face recognition systems like DeepFace or FaceNet, which I'd be interested to see if there's any correlation between the embeddings they produce if a simple PCA model works so well)

Also don't be so dismissive, we have a strong enough empirical and intuitive understanding of CNNs that we're able to make thoughtful improvements over time. In fact the insight behind the ResNet paper was noticing that adding layers doesn't improve performance and that training error actually degrades as layers are added – the solution to this was to construct the network so that it learns residual mappings that only modify the input rather than completely transform it. The whole point of that paper was solving this degradation problem so they could use some ridiculously deep architecture like a 150-layer network to get better results.