[tejask]

In summary, the most interesting part for the general audience might be the following question -- can we learn a 3D rendering engine just from images or videos without any hand-engineering?

Apart from the interesting applications for computer graphics (like rendering novel viewpoints of an object from various viewpoints), this can also be directly used for vision applications. This is because computer vision can be thought of as the inverse of computer graphics.

Goal of computer graphics: scene description -> images

and

Goal of vision: images -> scene description.

Therefore, training a neural network to behave like a graphics engine is interesting from both these perspectives. We are a LONG way from even scratching the surface.

[perdunov]

How long has this idea of making a 3D engine from conv nets been researched?

[tejask]

To the best of my knowledge, not much at all. It is an open question. Besides, a feedforward net is not going to be enough.