[westoncb]

> Composite those things together manually and add a style transfer you'll get similar results to DALL-E as that is what it is doing more or less.

If you try actually doing this it will be trivial to see that this assertion is incorrect.

1. The way in which the elements of the images are integrated together is deeper than the level of style. For instance, see the image in the top row, second column: it has integrated the blue bird wings onto the man, not only simply grafting them on, but giving the appearance of their being draped on like a cloak, partly behind and partly in front of him (+ it's consistent with the man's posture and the rays of light to evoke a certain coherent cultural idea/image). You might be able to integrate multiple images (of man, bird, rays etc.) together and style transfer to arrive at a poor approximation of this—but even then, the decision to place the elements together in such a way would require creativity on your part.

2. The one example set of of trial images (generated from the phrase "expressive painting of a man shining rays of justice and transparency on a blue bird twitter logo") is one of the easiest among the full group to pick its various elements apart; if you try this thought experiment with the others in the thread, you'll see this idea is by far insufficient.

[recuter]

Good, finally. Yes, exactly - this is the most interesting aspect of the whole thing.

> the decision to place the elements together in such a way would require creativity on your part

I strongly suspect that's because it found similar compositions in its training set. So what exactly is going on here is fascinating.

Did it learn compositing? Is that why the image output is now much more stable? Or is it mearly finding similar artwork and competently recreating/mimicking existing compositions from different building blocks? So now we can not only transfer styles but also transfer compositions. That could be the beginning of something useful. Instead of a text prompt I'd give it my crappy doodle and it will respond with an improved/different one that is comparable (also a great way to steal tho).

And of course I picked the one that is easiest to tease apart where it is most evident so people will see what I mean.

> if you try this thought experiment with the others in the thread, you'll see this idea is by far insufficient

That depends on your imagination and your artistic eye I guess. Even if somebody could do that they certainly couldn't make you believe them. That's the accomplishment.

Neither one of us can prove it one way or the other so long as the model is a black box. And certainly so long as we don't have direct access to openai but just to curated examples.

[samvher]

On (2), so this part is where I wonder: no-one has "expressive painting of a man shining rays of justice and transparency on a blue bird twitter logo" as their twitter bio. So are the "happy sisyphus" images generated from "happy sisyphus children's style", or are they generated from something more like "a person carries a large ball in a mellow image in the style of a pixar cartoon"? To me there is a huge difference between these things: how much of the context is inferred from the bio, and how much from what's provided in the prompt? (Does DALL-E 2 know about the story of Sisyphus or is that part filled in?)

[recuter]

In the video accompanying the paper they gave the example of "tree bark". Do we mean the bark of a tree or a dog barking at a tree?

So I reckon with "happy sisyphus" it breaks it apart into discrete vectors as a first disambiguation step and in this case resulting in two distinct queries.

Happy returns all kinds of image results.

Sisyphus returns the same kind of image results over and over.

A man rolling a boulder up a hill. Thus it can learn the concept of "sisyphus" on the fly as it would return:

man 95% boulder 90% hill 80% etc

Over a range of images.

So it must be Man+Boulder+Hill. That's its scene cue. That's what CLIP doodles initially. That's the "find me similar images step".

Happy is the style cue.

That's how "happy sisyphus" expanded into "a person carries a large ball in a mellow image in the style of a pixar cartoon"

Why specifically the Pixar style? One of several variations it tried, selected by a human.

The thing we don't know is whether the Pixar styled image is composited from the existing images in its training set. In other words whether this can be reversed.

That character looks familiar tho. I think it is plagiarizing.

Here is another observation: the boulder is not round, it reminds me of one of the Platonic solids. I don't think that's a coincidence, heh.

[westoncb]

You're asserting a bunch of things about how it works that have no basis in reality. If you want to be able to comment on this stuff with any accuracy, read the research they've published.

[westoncb]

They are generated from e.g. "happy sisyphus". My understanding is there are separate additional controls for style (though it's flexible enough you could give hints in the text, too, which is I gather where the "expressive" word fits in).