If your thinking is along the lines “if jodrellblank can’t convince me that a superintelligence is plausibly dangerous, off the top of their head, before my attention span wanders, then AI can’t be dangerous” that’s a really weak plan for protecting humanity.
Say, pops up on the US or Kremlin computers and blackmails or bribes or threatens someone to launch the nukes? Say, places orders with a lab to build some robots which then become the AI’s body and it builds nukes itself. Say, placed orders for custom biological things using stolen money which turn out to be extremely virulent and then it doesn’t need nukes. Say it finds out an effect to worsen global climate change and uses it to make earth uninhabitable on a short timescale, and doesn’t need nukes.
Social engineering, for a start. Everyone with a secret will be a target.
Half the world will follow stupid leaders. An AI would run rings around your average voter. I'd be surprised if it can't run rings around basically every voter given time.
We're also creating various kinds of robots, so the thing could just plug itself into the network it wants.
That's 5 minutes. An AI would be smarter than me and have a lot more time every second to think up better ways, and A/B test them on millions of people.
All it needs is one guy thinking "Ah, what's the worst that could happen? It promised me a million dollars."
Some contractor creates an internal webhook for testing the silo door that later gets accidentally hooked up to the full launch sequence and - during a billion dollar DoD mainframe upgrade project (to finally get away from COBOL) - gets exposed to the internet. MAD doctrine does the rest.
That is how the Anthropocene ends and the age of the machines begins. So say we all.
There are a number of hair-raising stories tucked away in there.
Paired with knowledge of things like Stuxnet (https://en.m.wikipedia.org/wiki/Stuxnet), the broken arrow list fills with worry for what might happen with nuclear weapons systems in our modern, hyper-connected world.
Say, pops up on the US or Kremlin computers and blackmails or bribes or threatens someone to launch the nukes? Say, places orders with a lab to build some robots which then become the AI’s body and it builds nukes itself. Say, placed orders for custom biological things using stolen money which turn out to be extremely virulent and then it doesn’t need nukes. Say it finds out an effect to worsen global climate change and uses it to make earth uninhabitable on a short timescale, and doesn’t need nukes.