I may be understanding Seurat wrong but I think its more than just that. I think they bake a surface light field and do some form of image-based rendering against a much lower-poly version of the mesh.
Correct: it generates a surface light field from a set of RGBZ images that sample the scene surfaces. In turn, that's simplified into a set of alpha-textured quads that very accurately represent the scene from a specified headbox.
View frustum culling wouldn't have been enough to render this stuff (https://www.roadtovr.com/preview-google-seurat-ilm-xlab-mobi...) in VR on a mobile GPU. :-)