[Const-me]

> That seems very doable with some GPU code.

This seems about as hard as digitally moving eyes.

I think the main source of artifacts is going to be lighting and reflections. Specular color or reflections are only possible to see when light, surface position and normal, and observer are arranged in a specific way. If you have 2 or more cameras positioned elsewhere, there's no way to find out what color is visible to another camera in the center.

Modern AI can try to guess, but fundamentally there's no that info anywhere in the video. It can assume the object surface is made of small count of uniform materials, and extrapolate materials across picture and across frames, but this gonna fail too often for biologicals subjects like people.

[TaylorAlexander]

Moving eyes means making decisions about human behavior, which is hard. Any weirdness will be very detectable. Just doing a 3D reconstruction with multiple cameras is more established field.

[Const-me]

> Just doing a 3D reconstruction with multiple cameras is more established field

Yes, but that alone is not enough. You can indeed reconstruct 3D after spending enough resources, but that won’t help you finding out which color the camera is going to see, because of these reflection issues. Human eyeballs are very reflective. Even if you approximate them with spheres and distort the reflections accordingly, next subject will wear eyeglasses, the reflecting shape of these is arbitrary, you have no chance of doing that accurately enough.

The worst-case example is a person wearing eyeglasses which are completely flat on the outside. No matter how many cameras are around the screen, none of them will capture what would reflect in the eyeglasses for a missing camera at the center of the screen.

I think people will eventually solve that, not with AI postprocessing, with hardware. You can place a camera behind center of the screen, and split time between display and camera. For example, you light the display for 10ms, and for the next 6.66ms you turn off the display and instead read data from the camera. This will get you 60Hz of both display and camera.