Real time novel view synthesis is already possible. In fact, I recently saw someone demo a version that ran in WebGL2 shaders on smartphones, so it's not terribly taxing.
Thanks very much for the reference. I'm curious about this but unfortunately not enough to keep up with the nooks and crannies where it's being worked on!