[SonOfLilit]

I'm afraid global warming and plastics in the ocean might make our quality of life much worse, but if I'd had to bet on what risks human extinction, "bad people with superweapons" would be higher, and "autonomous agents a bit smarter than any human that are built to compete with it" higher still, if they were options on the page.

And how reasonable is that we would have autonomous agents a bit smarter than humans in the nearish future? Very low, I'd say, about the same as the odds I'd give a year ago for GPT4? Not odds I'd be happy to take for the _destruction of humanity_.

[slowmovintarget]

Right. The problem is not super-intelligent autonomous agents. The problem is not-quite-smart-enough autonomous agents acting on foolish instructions.