[campground]

I tried what you suggested for a bunch of the twitter bios and found nothing except links back to this thread. I also reverse image searched a bunch of them to see if DALL-E was just kind of pasting together large chunks of images, but never found anything close. I do think you're being dismissive but please post any examples of what you mean. I'm a skeptic and have been waiting to find out that this is just a glorified parlor trick, but so far it seems like DALL-E is doing everything the authors claim, which is remarkable.

[joshcryer]

It's fascinating how in our hubris we were thinking that art would be the last thing for AI to tackle, but it appears to be the first (Sam Altman made a similar statement on the launch of DALL-E). Which makes art more meaningful to me, for some reason. There's something in the billion parameters and exabytes of data that this neural net had to process and it was so ... easy. Natural. Because it is us. It is our expression. Our creativity. Our outpouring of data, and all it is doing is reflecting us. It's beautiful.

[boppo1]

I'm an amateur painter and AI hasn't even kissed high art yet IMO, although it's nominally good at amateur illustration.

I'm going to have to write up a piece on this sometime, my argument is a little too involved for an HN post. But the gist is that the heart of what a fully trained painter does is make personal choices. A quick and dirty example of the difference:

Suppose you train an AI on Picasso's pre-1901 pieces. It's not going to decide it's time for a blue period.

[mensetmanusman]

That’s because the entire corpus of art is stored in the neural network weightings as memory. It’s built to imitate human art by optimizing towards these weightings.

[recuter]

There is something about clouds. Facial recognition software frequently finds faces in clouds just like we do.

This thing will kill art dead.

[boppo1]

Check my post prior to this one. Art will be fine.

Well, representational art. I'd like to say post-abstract expressionists are at risk, but they still have their admirers convinced they're wearing clothes, and there's no indication those idiots will ever change their minds.

[recuter]

I agree with your post. Good luck convincing every child with a big Dall-E button on their iPad that they are not in fact Picasso. I mean, just look at this thread. And these are supposed to be adults.

[rndphs]

Yeah I just tried google image searching to find something like the pikachu photo from https://mobile.twitter.com/gottapatchemall/status/1511777860...

But I can't find anything close to the realism that DALL-E 2 achieved here.

[recuter]

There was an abomination of a live action Pikachu movie some time ago. When I google "realistic pikachu" I get images exactly like this from the movie but not gross.

In fact this photo is exactly what you get when you photoshop the face of an ugly chihuahua unto a Pikachu plushie head and add a yellow brushed hamster body. And a cape. Literally that is what you're looking at.

It understood your prompt and amalgamated the right source photos into this nightmare fuel. Jesus wept.

[chrisco255]

Yeah, it's still impressive to be able to imitate those styles and add a blue cape that didn't exist in the movies, along with chihuahua eyes. It also appears to be higher definition than Detective Pikachu CG. I'm curious if you could do the same for all 150 original Pokemon, even those for which realistic CG representations don't exist. Would it be able to take the cartoon version of Farfetch'd or Psyduck or a more obscure one and achieve the same realism, without the reference from the deep dataset?

[rndphs]

Well to my eye it's realism beyond anything that I could find. Mind you I didn't search for that long so there might be something there if I was to delve deeper.

I am pretty familiar with photoshop, and while I'm not an expert, I would find making something like this really difficult. Anything is possible with photoshop, but some things are very hard.

[ShamelessC]

> In fact this photo is exactly what you get when you photoshop the face of an ugly chihuahua unto a Pikachu plushie head and add a yellow brushed hamster body. And a cape. Literally that is what you're looking at.

i guess some people are overhyped, but it's cool that this can do that. Previously, it took a trained human.

[recuter]

If this is the exact image you wanted and are entirely satisfied for it, great. But what people are reacting to is that it is outputting interesting images at all.

What are you going to do with this cape wearing realistic Pikachu that is actually a picture of a hamster?

Typically the trained human has something specific in mind. And if the client isn't satisfied they will torture them with countless requests for adjustments. So right now this is of limited use.

To me what is far far far more interesting is that Dall-E possibly understands the concept of what a Pikachu is supposed to be. That is downright creepy, and fascinating. I suspect that this visual aspect to things after people get over the clipart generation might find more functional utility as a way to see through the "model eyes" so to speak. To visualize the model itself. That could unlock a lot of doors in how training is done.

Maybe in the future you could train it on textbooks and prompt it for a picture of a molecule. Now that would be something. Especially if you start feeding it data from experiments.

[fastball]

> Typically the trained human has something specific in mind. And if the client isn't satisfied they will torture them with countless requests for adjustments. So right now this is of limited use.

Confused as to why you think you cannot do this with DALL•E?

[recuter]

I don't want to be dismissive of Dall-E itself or its authors. Just the implications that this changes everything or how it is much more than it really is.

https://twitter.com/nickcammarata/status/1512123067803344899...

Prompt: "expressive painting of a man shining rays of justice and transparency on a blue bird twitter logo"

You have to break the concepts up apart (which is one of the things Dall-E improved on).

As such: "expressive blue bird"

In google image search, type clipart, and I even get pill tags to further narrow it down to illustrations for animal paintings and so forth. Google's classifier knows the concept of a "blue bird" and expressionism too.

https://www.google.com/search?q=expressive+blue+bird&tbm=isc...

The same for "ray of light". In fact the top results there I get pngs of sun beams on a transparent background. Which is perfect.

Neither the birds nor the rays of light in the pictures it produced are truly its own creations but lifted from bits of pictures in its training set. I bet you could find the exact bird from the second row online in many places for example. It just won't be blue or stylized.

Composite those things together manually and add a style transfer you'll get similar results to DALL-E as that is what it is doing more or less.

[westoncb]

> Composite those things together manually and add a style transfer you'll get similar results to DALL-E as that is what it is doing more or less.

If you try actually doing this it will be trivial to see that this assertion is incorrect.

1. The way in which the elements of the images are integrated together is deeper than the level of style. For instance, see the image in the top row, second column: it has integrated the blue bird wings onto the man, not only simply grafting them on, but giving the appearance of their being draped on like a cloak, partly behind and partly in front of him (+ it's consistent with the man's posture and the rays of light to evoke a certain coherent cultural idea/image). You might be able to integrate multiple images (of man, bird, rays etc.) together and style transfer to arrive at a poor approximation of this—but even then, the decision to place the elements together in such a way would require creativity on your part.

2. The one example set of of trial images (generated from the phrase "expressive painting of a man shining rays of justice and transparency on a blue bird twitter logo") is one of the easiest among the full group to pick its various elements apart; if you try this thought experiment with the others in the thread, you'll see this idea is by far insufficient.

[recuter]

Good, finally. Yes, exactly - this is the most interesting aspect of the whole thing.

> the decision to place the elements together in such a way would require creativity on your part

I strongly suspect that's because it found similar compositions in its training set. So what exactly is going on here is fascinating.

Did it learn compositing? Is that why the image output is now much more stable? Or is it mearly finding similar artwork and competently recreating/mimicking existing compositions from different building blocks? So now we can not only transfer styles but also transfer compositions. That could be the beginning of something useful. Instead of a text prompt I'd give it my crappy doodle and it will respond with an improved/different one that is comparable (also a great way to steal tho).

And of course I picked the one that is easiest to tease apart where it is most evident so people will see what I mean.

> if you try this thought experiment with the others in the thread, you'll see this idea is by far insufficient

That depends on your imagination and your artistic eye I guess. Even if somebody could do that they certainly couldn't make you believe them. That's the accomplishment.

Neither one of us can prove it one way or the other so long as the model is a black box. And certainly so long as we don't have direct access to openai but just to curated examples.

[samvher]

On (2), so this part is where I wonder: no-one has "expressive painting of a man shining rays of justice and transparency on a blue bird twitter logo" as their twitter bio. So are the "happy sisyphus" images generated from "happy sisyphus children's style", or are they generated from something more like "a person carries a large ball in a mellow image in the style of a pixar cartoon"? To me there is a huge difference between these things: how much of the context is inferred from the bio, and how much from what's provided in the prompt? (Does DALL-E 2 know about the story of Sisyphus or is that part filled in?)

[recuter]

In the video accompanying the paper they gave the example of "tree bark". Do we mean the bark of a tree or a dog barking at a tree?

So I reckon with "happy sisyphus" it breaks it apart into discrete vectors as a first disambiguation step and in this case resulting in two distinct queries.

Happy returns all kinds of image results.

Sisyphus returns the same kind of image results over and over.

A man rolling a boulder up a hill. Thus it can learn the concept of "sisyphus" on the fly as it would return:

man 95% boulder 90% hill 80% etc

Over a range of images.

So it must be Man+Boulder+Hill. That's its scene cue. That's what CLIP doodles initially. That's the "find me similar images step".

Happy is the style cue.

That's how "happy sisyphus" expanded into "a person carries a large ball in a mellow image in the style of a pixar cartoon"

Why specifically the Pixar style? One of several variations it tried, selected by a human.

The thing we don't know is whether the Pixar styled image is composited from the existing images in its training set. In other words whether this can be reversed.

That character looks familiar tho. I think it is plagiarizing.

Here is another observation: the boulder is not round, it reminds me of one of the Platonic solids. I don't think that's a coincidence, heh.

[westoncb]

You're asserting a bunch of things about how it works that have no basis in reality. If you want to be able to comment on this stuff with any accuracy, read the research they've published.

[westoncb]

They are generated from e.g. "happy sisyphus". My understanding is there are separate additional controls for style (though it's flexible enough you could give hints in the text, too, which is I gather where the "expressive" word fits in).

[joshcryer]

I think your last line is what stands out more than anything. You've just described creating something without "compositing those things together manually."

Note that in that example the "twitter bird logo" is actually expressed in 6 out of all of those images. Look for the small bird, that looks like the Twitter logo. It's there. It's doing the thing.

[recuter]

The prompt is actually "blue bird twitter logo".

Nothing is expressed. Find yourself a blue bird in an expressionistic style, go to google image search and give it the url. Click on tools -> visually similar.

Enjoy an endless supply of things to plagiarize. In the middle picture of the second row you can clearly see how several pre-existing images are sharply cut off before being re-blended.

[ShamelessC]

Same thing going on here as in your other comments.

Tech like CLIP, GPT-3, DALL-E, etc. are indeed nearing the sophistication (w. caveats around outliers and harmful outputs) of Google search.

It took a lot of people to create Google search. It took precisely one training run for DALL-E 2 to create this.

edit: Removed toxic comment.

[recuter]

No, don't get me wrong. I think DALL-E is very interesting and a potentially useful tool and have nothing against the tool makers.

The tool wielders however.. I think are overyhyping this to say the least. And focusing on the wrong bits. It isn't sentient and it is not making art. But teasing apart how it is deriving these images might shake out serious advancements.

[ShamelessC]

Fair enough I think we are in agreement.

[boppo1]

> Prompt: "expressive painting of a man shining rays of justice and transparency on a blue bird twitter logo"

Yeah, weak results. None of the men look anything like Elon Musk. /s