[JoshTriplett]

> The same things could be done by you or me using the internet or books though, why does the model make it different?

Imagine two worlds. In one world, everyone has a button that ends the world, which is badly labeled and may also press itself at any time. In another, people who have gone through a substantial amount of effort and dedication to learn something extremely difficult, also understand that they could apply that knowledge towards bad ends. Which world exists for longer?

> what kind of prompt are we writing where a model accidentally creates a contagious disease or self-propagating worm as one of its goals?

Given a sufficiently powerful model? Any prompt that could be done better by seizing additional computing power, or preventing the operators from turning it off. https://en.wikipedia.org/wiki/Instrumental_convergence