I'm definitely not an expert, but I think the way to go is to use an algorithm that can find a specific face in the latent space. If the latent space has enough dimensions (trained with a big enough set) it should already contain your face. The problem is to find it.
Thanks for the reply! I just want to add a little more details. As long as the training dataset is diverse enough, the trained GAN generator should be capable of generating images that are similar to the training dataset. Therefore there is no need to retrain the GAN. However, to "embed" a given face image to the latent space, we need to either use the optimization based method https://arxiv.org/abs/1702.04782 mentioned in "Geee"'s post, or train a encoding network. The encoding network would be much easier to train than the decoding/generator network because we do not need to use adversarial loss. I would love to include this functionality into my model once I got a job offer and have some spare time :)