[TiredGuy]

The state of the art of 3d pose estimation and pose transfer from video seems to be pretty accurate. I wonder if another approach might be to infer a wireframe model for the character, then tween that instead of the character itself. It would be like the vector approach described in the article but with much, much fewer vertices, then once you have the tween, use something similar to pose transfer to map the most recent character's frame depiction to the pose.

Training on a wireframe model seems like it would be easier, since there are plenty of wireframe animations out there (at least for humans) you could use and remove in-between frames to try inferring them.