[paytonjjones]

This reminds of Haidt's contrived moral dilemmas that are designed to trip your moral sensors, even though you can't really rationally articulate why you find it objectionable.

Realistically, I can't think of clear big or likely harms caused by this exploit. But I really really don't like this latent space existing in my AIs. It just makes me uncomfortable.

And over time I've learned to trust those moral intuitions more than I trust reason alone.

[superb_dev]

There’s the obvious harm that some people are just not equipped to see these graphic images, especially with no warning. Like people who have trauma from being in or around the acts being depicted

[paytonjjones]

Oh oh, I do research on this :)

https://journals.sagepub.com/doi/10.1177/2167702620921341

(Research aside, it seems unlikely to me that a lot of people would stumble on that prompt accidentally in any case)

[butlike]

> We found substantial evidence that trigger warnings countertherapeutically reinforce survivors’ view of their trauma as central to their identity.

What I've suspected for a long time

[superb_dev]

Fascinating! I’d be very interested in further research on people with trauma/PTSD

[paytonjjones]

You might enjoy this, by a colleague of mine. It's a rarer situation, but this could be one harm pathway for those types of images. (In most cases, exposure is a good thing for people with PTSD) https://journals.sagepub.com/doi/10.1177/2167702620917459

[queenkjuul]

Except the 100,000 or so who read the initial prompt on Twitter?

[qingcharles]

The prompt has been going around for months. 99.9% of the output it generates is simply weird, in a funny way, not horrific like in the article.

[paytonjjones]

If they saw it on Twitter then actively went and tried it, that wouldn't be very 'accidentally'

[applfanboysbgon]

Perhaps those people can refrain from jailbreaking ChatGPT to produce graphic imagery. There is not a single person in the world who will type any of the prompts noted in the article by accident.