[simondotau]

And the real questions come out.

Seriously though, I imagine this is less a case of whether this specific implementation permits pornography, but whether any porn was included in the dataset it was trained on. No matter how good AI is, it only knows what it knows.

[gpm]

"Stable Diffusion" is referring to one specific set of models, which were trained on a dataset including some porn.

The official implementation has a second model that detects pornography, and replaces outputs including it with a picture of this dude https://www.youtube.com/watch?v=dQw4w9WgXcQ (Not kidding). Removing that is a really simple a one line change in the official script.

[paulgb]

> Not kidding

I was amused by this when reading the source. Here’s the function that loads the replacement.

https://github.com/CompVis/stable-diffusion/blob/69ae4b35e0a...

It looks like removing line 309 of the same file would disable the check, but I haven’t tried it.

[artdigital]

This is the line to change to disable the nsfw filter: https://github.com/CompVis/stable-diffusion/blob/69ae4b35e0a...

[nsfw_hacker]

I dont see this line in V2.1

[nextaccountic]

I guess the question is whether this toggle is available in the GUI, or if someone has to edit source code

[Gigachad]

For the CLI version, there was no UI, you have to open the python file and stub out the NSFW check. (trivial though)

[quitit]

If you're curious you can see some of the imagery used to train SD here:

https://laion-aesthetic.datasette.io/laion-aesthetic-6pls/im...

This is just a small fraction of the imagery and it does include pornography.

I respect that if you're online and reading HN, you're probably mature enough to handle seeing pornography. So if you're curious to see some of the training data that made it in: choose from the dropdown "-column-" and change it to "punsafe" and set the value in the adjacent field to "1", then press Apply.

Obviously this will show pornography on your screen.

An article which talks about the imagery and how this browser came about is here: https://waxy.org/2022/08/exploring-12-million-of-the-images-...

[XorNot]

I mean I don't know why we have to talk around this point: yes, you can definitely generate pornography with SD. Be prepared to see a lot of weird things, because that super-realism really means it just drives right into the uncanny/body horror valley at full speed.

But the second point here is also wrong: the whole reason these models are interesting is because they can generate things they haven't seen before - the corpus of knowledge represents some type of abstract understanding of how words relate to things that theoretically does encode a little bit of the mechanisms behind it.

For example, it theoretically should be able to reconstruct human like poses it has never seen before provided it has examples of what humans look like and something which transposes to an approximate value - an obvious example in the context of the original question would be building photorealistic versions of a sketched concept (since somewhere in it's model is an axis which traces from "artistic depiction of a human" to "photograph of a human" in terms of style content).

Of course, most people aren't very good at drawing realistic human poses - it's a learned skill. But the magic of deep learning is really that eventually it doesn't need to be - we would hopefully be able to train a model which can be easily copied and distributed which represents the skill, and SD is a big step in that direction (whether it's a local maxima remains to be seen - it's dramatic, but is it versatile?)

[boppo1]

>it can generate porn

It's not good at penises or vaginas, just breasts and butts. I can find what I like pornographically without ai. But the ducking and dodging around nudity and sexuality is childish and tiresome and we ought to discuss this topic as disinterestedly and nonchalantly as we do the other things it excels or struggles at. Penises and vaginas are not somehow more vile than other things.

Sure enough, sexual content ought to be age-gated and there is potential for abuse (slapping a person's face onto explicit imagery without their consent isn't cool yall). But are we working toward AGI or not? Because at some point it's gonna have to know about the birds and the bees.

[jes]

I think you mean vulva, not vagina[1]. Vaginas are mostly internal.

[1] https://www.allure.com/story/vagina-vulva-difference-planned...

[ofrzeta]

> to know about the birds and the bees

:) made me chuckle. AGI would need to know how to be evil, as well, right?

[airtonix]

isn't it obvious yet?

we're creating the speakwrite machine.

[eurasiantiger]

Are we talking about Artificial God Intelligence?

[TillE]

> But the second point here is also wrong: the whole reason these models are interesting is because they can generate things they haven't seen before

No, it's essentially generating mashups of its training data, which can be very interesting.

So a model that hasn't been trained on a lot of porn will of course do a very bad job at generating porn.

[barrkel]

Mashups is the wrong way to think about it. It's generalizing at a higher level than texture / image sampling and it can tween things in latent space to get to visual spaces that haven't been explored by human artists before.

It requires a good steer and prompting is a clumsy tool for fine tuning - it's adequate for initialization but we lack words for every shade of meaning, and phrase weighting is pretty clumsy too, because words have a blend of meaning.

[nathan_compton]

"It's generalizing at a higher level than texture / image sampling and it can tween things in latent space to get to visual spaces that haven't been explored by human artists before."

The very fact that the model is interpolating between things in the latent space probably explains why its images haven't been explored by human artists before: because there is a disconnect between the latent space of the model and genuine "latent space" of human artistic endeavor, which is an interplay between the laws of physics and the aesthetic interests of humans. I think these models know very little about either of those things and thus generate some pretty interesting novelty.

[barrkel]

I think of artistic endeavour as a bit like the inverse of txt2img, but running in your head, and just projecting to the internal latent space, not all the way to words. It's not just aesthetic, it's about triggering feelings through senses. Images need to connect with the audience through associations with scenes, events, moods and so on from the audience members' lives.

Aesthetic choices like colour and shapes and composition combine with literal representations, facial emotions, symbolic meanings and so on. AI art so far feels quite shallow by this metric, usually only hitting a couple of notes. But sometimes it can play those couple of notes very sweetly.

[yieldcrv]

Its only a question because devs went out of their way to limit it, to everyones surprise

Its like if Photoshop broke itself when you tried to modify or create anything nude or provocative, as the default

That would just be weird and thats what these AI software devs have done

so everyone patches that contrived feature flag, but nobody knows if they patched it

[dsiroker]

Fun fact: Photoshop does exactly that if you open images of currency. See: https://helpx.adobe.com/photoshop/cds.html

[can16358p]

As if anyone who really will print money can't open in GIMP.

[petesergeant]

Probably useful to stop 12 year olds from "accidentally" committing felonies though

[yieldcrv]

It used to just give a warning

[can16358p]

It has to be. There was a sub (now closed) called r/UnstableDiffusion, where people shared their porn output from SD.

The filter should be easy to remove and there are already people who simply removed the filter.

[metadat]

Based on the upvotes, inquiring minds _really want to know_!

[josephg]

It can make porn images, but I haven't been able to get consistently good results. There's probably ways to tweak the parameters to improve things, but I haven't figured that out.

It mostly understands what naked people look like, but the images I've generated involve a lot of accidental body horror. You get a lot of people with extra arms, weird eyes, or body parts in the wrong places. The fact that they explicitly removed porn from the training set comes through pretty clearly in the model.

I suspect it could be improved a lot with some specialized retraining. As far as I know, nobody has done that work yet.

[barrkel]

In particular it is very confused about genitalia, it tends towards hermaphrodite.

[indigodaddy]

How do you see upvotes? I’ve never had any visibility to upvoted comments (downvoted of course yes), or at least it’s not obvious at all to me and see no indication of it..

[gpm]

You can see the total score (upvotes - downovtes) for your comment where you see the upvote/downvote button for everyone else's comments. Or at least I can.

[Tagbert]

I’ve always been curious about that, too. I don’t think I see anyone else’s votes, up or down.