[modulus1]

And they appear to have some way to fill in the part of the photo occluded by the cut-out objects.

[bhouston]

They never seem to mention that in the paper, at least not prominently as I of course skimmed it today. But Photoshop already has a built in tool for this, so I guess they can just use the standard methods that seem to work fairly well.

[smackfu]

"We compute a mask for the object pixels, and use this mask to inpaint the background using the PatchMatch algorithm [Barnes et al. 2009]"

PaintMatch algorithm: http://gfx.cs.princeton.edu/pubs/Barnes_2009_PAR/index.php

[bhouston]

Yes, that is the one that Photoshop has adopted and renamed " Content Aware Fill"! Details: http://www.adobe.com/technology/projects/patchmatch.html

[cbhl]

Judging from the YouTube videos, the novel part is that they can fill out the part that is occluded from the photo (either using textures from the 3D model, or by using InPaint) because they refer to earlier work that already lets you cut out and manipulate the objects using 3D models.

[thearn4]

Yep, they probably used an off-the-shelf method for that. It's often called image interpolation or "inpainting" (https://en.wikipedia.org/wiki/Inpainting).