Here's a paper around synthesizing human faces. It includes inpainting http://www.faculty.idc.ac.il/arik/seminar2009/papers/VisioFa...
https://arxiv.org/pdf/1604.07379.pdf this uses a GAN to inpaint with arbitrary data. This is probably a couple of iterations from being easy to implement, as training GANs efficiently and accurately is still a technical challenge.