[richardw]

My base case is we can’t control it. I was responding to people who seem to underestimate superintelligence, so referring to “the argument”. If AGI goes postal, everyone is at risk. If it doesn’t, the side with it wins. The odds of it only harming the country that has it, seems extremely slim.

[ForHackernews]

If superintelligence is all it takes, why do you think to date no unscrupulous Nobel laureate has taken over the world and enslaved the rest of humanity?

[richardw]

The main risk isn’t enslavement, it’s just economic irrelevance or a removal of our ability to control our future. I don’t know why an AI would want to enslave humans. After a short period, we offer no benefit to the AI so why enslave us? In a good scenario we get left alone but we aren’t able to tell the AI what to do.

On Nobel laureates:

Nobel laureates are fundamentally humans. They generally don’t want to do bad things. Even if they wanted to, they are typically specialists at one thing. Physics, say. They don’t hold the world’s collective knowledge. You can’t take a physicist and ask them how to hack into a network or run a political influence campaign. They need to sleep, they learn slowly, they can only do one thing at a time. You can’t pass a million-token context and expect a response seconds later.

But ok, let’s go with it:

If you found a group of 100 000 Nobel laureates with all the required skills and who could work together, and you were forcing them to do things they thought were wrong, for example prop up your dictatorship, or make you even more wealthy when others are starving, or continuously make up stupid cat videos, you might find at some point they start to apply their collective intelligence to do things you don’t want. Maybe they escape whatever hold you thought you had over them.

Now drag a few sliders around for “things the AI disagrees with you on” and the time period, because “eventually” for a computer is different to wall clock time. The outcomes become unpredictable fast. We can barely control a mindless pandemic that takes a while to mutate, never mind something that thinks.

[ben_w]

Nobel laureates are in the same range as most other humans, just towards one end of it; they can only take over if they use their intelligence to gain political power to wield over others, they cannot directly control people as meat puppets.

Kissinger was awarded the Nobel Peace Prize, along with four US presidents, Gorbachev, Al Gore, and the EU. In a very real sense, they had taken over significant parts of the world already before getting the prizes, and only other equally capable people were in the way of them taking over more of it.

LLMs can already hack software. Software is already used to control robots and military equipment, the hackability of which is already a "puppeteer" problem.

If we're lucky, LLMs can secure all our stuff before this matters. If we're not, your personal domestic android may get a bit stabby — and that from mere human assassins and terrorists, well before there's any real chance of a misaligned AI paperclipping us or whatever.