[Animats]

It's not random noise. Look at the images in the paper. Horizontal lines, vertical lines, snakeskin patterns, Minecraft textures. Examples of miscellaneous surface patterns, in other words.

Back before deep learning, people used to make recognizers for features like that as a lower level of feature recognition. Now it's expected that features will be derived automatically from real imagery. This is kind of a return to that level.

A useful training set might be a big texture library used for game development or animation. Those are easily available.

[jswulff]

That would indeed be an interesting thing to try, use real data, but only in terms of textures - so effects like occlusions, perspective, etc. would not be present.

I would expect it to be somewhere in the ballpark of our StyleGAN images, which also look very "textural", but lack these effects that are an result of imaging the 3D world. Interestingly, modelling these effects without realistic textures seems to result in worse performance - this is for example the case for images taken from CLEVR or generated from Minecraft, and both perform worse than the StyleGAN images.