[bhouston]

Neat stuff. I wonder if one could use a neural network to try to do prediction as well.

Human movements are super stereotyped in the face and you could likely predict with decent accuracy the next frame given the previous frames.

[tsumnia]

Check out a lot of DisneyResearchHub's videos on YouTube. They are regularly posting research on object/face tracking and projections.

That said as you mentioned faces are pretty stereotyped. We "solved" face mapping two decades ago with Tim Cootes and Paul Ekman's work. We're able to quickly map rough estimates using traditional haar cascade classifiers and Viola-Jones with AdaBoost. Neural Networks may help, but we have other solutions that also handle the problem with "relative" ease (ignoring lighting, occlusion, etc.)