[spookie]

You can think of a video as a set of images taken from the same camera from different perspectives. So, you can assume as if they're different cameras (with the added bonus that the camera intrisincs remain the same), and apply "multi camera" techniques. I'm assuming you "move" the camera for parallax and what not.

With this, you can retrieve depth information by correlating the difference in position of easily identifiable points, and recreate the scene as a mesh.

This is basically the basis of photogrammetry as I know it. AI solutions may help at various stages to speed up the process too.