[tome]

> they can be run autonomously and with very little supervision produce economic output that eclipses that of the most capable humans in the world.

What’s the evidence for this?

> Perverse instantiation of AI systems was accidentally demonstrated in the lab decades ago

What are you referring to?

[nopinsight]

By the time you find such evidence, it could already be close to game over for humanity. It’s important to get this right before that.

We already have significant warnings. See for yourself if latest models like Imagen, Gato, Chinchilla have economic values and can potentially cause harm.

[marvin]

Historical examples of perverse instantiation are everywhere: Evolutionary agents learning to live off a diet of their own children, machine learning algorithms attempting to learn gripping a ball cheating the system by performing ball-less movements that the camera erroneously classifies as successful, an evolutionary algorithm to optimize the number of circuit elements in a timer creating a timer circuit by picking up an external radio signal unrelated to the task and so on. Some examples are summarized here: https://www.wired.com/story/when-bots-teach-themselves-to-ch...

GP wanted a concrete example of a doomsday scenario of failed AI alignment, so in that context extrapolating to a plausible future of advanced AI agents should suffice. If you need a double-blind peer reviewed study to consider the possibility that intelligent agents more capable than humans could exist in physical reality, I don't think you're in the target audience for the discussion. A little bit of philosophical affinity beyond the status quo is table stakes.