The behavior at low strength is not what I would expect. There are very strong effects on the input image. Those effects are independent of the style applied.
That's a good observation. Theoretically, 0 strength should give you the content image. What I think is happening is that the algorithm is not trained to generate style vectors for photorealistic images, and so the mapping it learns from pixels to style vector doesn't work well. Maybe a term should be added to the loss function to generate itself when content and style image are equal.
Maybe train it 'backwards'? I mean take a painting [1] as the input and perturb it with a photographic image toward photo realism. [2]
[1]: Assumptions about the 'quantity of art' inherent in painting versus photography seem to be playing a role. Both are typically flat images and are computationally fungible. So maybe photo styled to photo might be a different training set. It's not that radically different from etching applied to photograph.
[2]: anyway, still an awesome project. What I imagined is something that could have subtle effects on a photograph as an artistic tool for photographers...because that's my bias.
It world be nice to be able to move the "Stylization strength" bar and see how the image changes, but it's probably too slow for a real time result :(.
What about making a short video or an animated gif? Is the transformation smooth? (Is the image at 53.2% similar to the image at 53.3%?) Or the texture is rearranged in a random way?
You can probably calculate some key points in the bar while the user is wandering, and then use them. This will kill the battery of the cellphones, so perhaps don't enable it by default.
Check out the Prisma app which has been doing this style of image processing using machine-learning for awhile now. The results of Prisma can sometimes yield mind-blowingly fantastic results.
Yes, a lot of other apps and websites have been doing this for years. But you have to upload your images to their server. The point of this is that it does all calculations locally in the browser, at the expense of some quality.
Damn dude this is next level! I was going to ask how you pay for GPU instance hosting, but it turns out my 8 year old laptop runs it pretty close to realtime
At one point my graphics card driver had to restart. Not sure what happened there as I was not able to reproduce the problem.