[malandrew]

Your comment makes me wonder if maybe we should start thinking beyond techniques designed to recognize faces and instead brainstorm how we would get a machine to learn not how to recognize but how to reproduce. I reckon going about it this way would be instructive on how to improve recognition since recognition and reproduction are two highly related activities.

[kastnerkyle]

Parts of machine learning used to be focused on density estimation / manifold learning for exactly this reason. It seems to be the case that supervised tasks (for sufficiently complicated and diverse problems e.g. ILSVRC, MSCOCO) implicitly learn some portions of what you would want from a density estimator. This is enough to generalize to new, but related tasks - see the huge body of work on pretrained VGG for {semantic segmentation, style, texture generation, depth/surface normal prediction}.

Learning a general density that is useful is quite difficult - you could be missing entire modes from the data distribution and never know it. You don't know what you don't know, after all. And there is still no clear (in my opinion) evidence that learning a density really helps generalize to a new task, especially when the end task is still a supervised one.

Explicit density modeling seems to help the most in semi-supervised / limited data settings, but any time you can bring some enormous but related labeled data to bear on your supervised problem it seems to win over any kind of density estimation tricks - compare trained from scratch nets for the tasks I mentioned, to ones exploiting pretrained VGG models.