[haraldurt]

There have been some fascinating steps taken towards that direction, e.g. [0] and its related work. There they take the approach of transferring input images back and forth domains (think, a street imaged in summer transferred to its winter manifestation and back or a synthetic GTA image to real-world and back are the examples in [0]). Doing this while simultaneously holding the semantic content of the input unchanged with a GAN-type strategy seems to be a way to coerce the neural net's internal representations to capture what we want them to instead of idiosyncrasies of the dataset.

[0] https://arxiv.org/abs/1711.03213