Oh yeah! I wrote a similar algorithm and constantly suffered similar problems. Constraining the latent space can be a major pain. Its hard to visualize but I think a lot of latent features may exist in non-continuous latent "pockets", meaning that the desired direction of the latent vector is dependent on your position in space.
I´m having a blast changing all the parameters. Really uncanney valley feelings. Some women come up very attractive, looking like Jessica Alba. How can I react to a face of a person that don't even exist?
On a second thought, I don´t mean that the controls behave randomly. I understand they affect related parts of the system. If you increase "baldness" on a woman's face, it will obviously increase the "male" factor and the "gray hair" factor. I understand that this faces are being generated from a continuous space. Fascinating.
Looking like Jessica Alba is not a coincidence. This model was trained on CelebA, a dataset of Hollywood celebrity face photos. Jessica Alba is almost certainly in the training set.