[pavlov]

To get an idea of how style transfer works, it can be useful to look at some less successful examples. Here's one from the same git repo:

https://github.com/lengstrom/fast-style-transfer/blob/master...

(Original photo: https://github.com/lengstrom/fast-style-transfer/blob/master... )

With the "Great Wave" painting as the network's style input, the limitations of the technique become more apparent. It's clear that a human painter would never render the Chicago skyline in this way: there are incongruent little waves on buildings' edges and all over the sky.

The antennas on top of the tallest tower are particularly revealing. The neural network just sees an area of higher local contrast, and has continued the same pattern that was applied in the sky at the top-right of the antennas but with more contrast applied. This doesn't make any sense for what's supposed to be a painting.

There's no intelligence here, "just" pattern matching that can do a brilliant illusion of creative variance on the right kind of content. ("Just" in quotes because it's still a great achievement.)

[pizza]

Now we need some kind of object recognition-enhanced version of style transfer that learns constraints on what "makes sense" given "sensible" labeled/captioned training examples!

[ebalit]

It has been done with manual segmentation [1]. And the results are mind blowing. There also a lot of work done on segmentation with neural nets [2, 3], so I wouldn't be surprised to see someone implementing this idea in the near future.

1: https://github.com/alexjc/neural-doodle 2: https://arxiv.org/pdf/1605.06211.pdf 3: http://mi.eng.cam.ac.uk/projects/segnet/