[oidar]

> If humanity goes extinct in the next few years because of unaligned superintelligence,

I've seen people claiming that this could happen, but I've yet to read any plausible scenario where this might be the case. Maybe I lack the imagination, could you enlighten me?

[drcode]

https://ifanyonebuildsit.com/

[YZF]

- AI smarter than any human.

- AI dominated the physical world. Robots, factories, etc.

- AI decides humans aren't contributing and/or wasting resources it feels should go somewhere else.

I mean not unlike humans causing extinction of other species?

[theamk]

That "etc" in "robots, factories, etc" is doing a lot of work here.

Factories, even fully robotic ones, heavily rely on humans to set up and maintain them. Moreover, the safety culture means there are tons of "disable" controls which can be triggered by any human and no machine can override.

Robots look impressive, but they cannot function without the humans either. Military kill-bots are likely the worst, but machines cannot repair or refuel them.

None of this is going to change in the "next few years".

[YZF]

Yes.

Robots can't function without humans because they're not super-intelligent. We already see quite capable humanoid robots. Those factories that rely on humans - they'll be converted to be operated by humanoid robots. By the super intelligence.

That's the hand wavy story. It's hard to dive into details in an HN comment but I'm happy to try and develop some of those details. You're saying that something much smarter than humans isn't going to be able to bridge the gap to the physical world. I'm not so sure.

EDIT: Another way to think about it is that if a god-like infinitely capable being took control of all our online digital systems including I donno Teslas, factory automation, power grid, any form of connected robot in the world, nuclear weapons launch systems, airplanes, whatnot, do they have any path to a sustainable "existence" without relying on humans. Or at least with us unable to detect and stop that. If the answer is no then we're probably safe. It's kind of hard to convince ourselves of that. Keep in mind that humans can also be manipulated to do work for this god just like spies/saboteurs e.g. are recruited online today and paid bitcoin to do some random master's bidding.

[theamk]

> Another way to think about it is that if a god-like infinitely capable being took control of all our online digital systems including I donno Teslas, factory automation, power grid, any form of connected robot in the world, nuclear weapons launch systems, airplanes, whatnot, do they have any path to a sustainable "existence" without relying on humans

ha ha ha no. Teslas run of energy and cannot refuel, a circuit breaker in factory pops and no robot can reach it (or maybe a roof leaks), and "any forms of connected robot" _either_ cannot walk the stairs or can maybe run for a hundred miles before running out of battery.

The "humans can be manipulated" is the only thing to worry about, and you don't need robots for that, other humans have been trying hardest to do it just fine for millennia. I guess it's up to you if you want to be afraid or not, but I am not seeing anything super special so far.

[cwillu]

I've yet to read any plausible scenario where stockfish defeats me, all the scenarios my friends come up with have obvious holes in the plays they suggest stockfish could make.