[codelord]

We live in a world where proven maniacs (e.g. Putin) have access to an arsenal of nuclear weapons that can essentially make the earth uninhabitable for all humans. That's a very real possibility (with no ifs and buts and maybes) that exists now and we have learned to live with it. Yet somehow the hypothetical scenario of a human exterminator super-intelligent AI is getting all the coverage.

[jodrellblank]

Making the Earth uninhabitable to all humans is bad news for Putin, a human who lives on Earth. He derives his power from hundreds of millions of people in Russia working, and his lifestyle from hundreds of millions of people around the world buying Russian exports so an attack on everyone is also going to harm him.

AI wouldn’t be human, it wouldn’t necessarily object to a world uninhabitable to humans, or be hurt by it. If it would be hurt by it AI doesn’t have a billion years of evolved survival instinct to preserve itself. A super intelligence has many ways to cause mass destruction that we can imagine but cannot yet do, whereas nuclear weapons are pretty much all or nothing explosions only. Nuclear assault could leave some remote places still inhabited, AI could make certain not to.

[_cs2017_]

Innumerable times quite reasonable political leaders have started wars that resulted in their own deaths; and sometimes resulted in the destruction of civilization as they knew it (most recently - World War I, which effectively put an end to the European monarchies).

This suggests that giving nukes to reasonable political leaders presents a high and concrete risk. Giving nukes to political leaders with crazy ideas presents an even greater risk.

Climate change seems to be another high and concrete risk.

AI escaping and taking over the world can be also considered a risk, but it's far more remote. It is similar to the risk of aliens attacking earth after detecting its TV and radio transmissions. Some people might argue that we should slow down the work on AI. Others may demand we also ban all TVs and radios on earth. Both proposals seem to be a bit of an overkill given how remote the associated risks are. Especially given that we have the far more pressing risks to address.

The concerns about AI risks are like a drunk driver going 90 mph on a country road and suddenly deciding to stop saying "goddamn" just in case Jesus returns and punishes those who invoked God's name in vain in violation of the Third Commandment.

[jodrellblank]

"AI escaping and taking over the world" is a phrasing which inverts the situation to make it sound much safer; a special effects explosion has to get everything right to be safe. If anything isn't right, people could get hurt by shrapnel, by pressure, by heat, by smoke, by nearby structures being weakened and collapsing, by that causing breakage of steam pipes or other secondary effects, by it triggering a chain of other fires or explosions, by nearby people reacting e.g. swerving the car they are driving, etc. There are few safe outcomes and many dangerous ones. If a baby elephant has to jump over you while you are lying on the ground without crushing you, it has to be graceful and precise in a way that baby elephants aren't. Maybe it will see you as a wobbly unsafe landing place and try to avoid you for that reason if you're lucky. If there was such a thing as a 'baby monster truck' well it would behave more like a monster truck with a brick on the accelerator. And your defense is to pooh pooh the idea that a vehicle "would escape and try to kill you" handwaving away all the times runaway vehicles kill people without any intent to do so.

CUDA (2007), capable GPGPUs (circa 2012), Attention Is All You Need paper (2017), GPT 2 (2019), ChatGPT (2022), OpenAI valued at ~$28Bn (April 2023), OpenAI valued at $80Bn (Feb 2024).

Computers use fewer bits to store a list of countries than humans use braincells to do the same, and they have an easier time of it. Computers do arithmetic much faster than humans. Computers use fewer logic gates to do arithmetic than humans use braincells to do arithmetic. I don't think we can take it for granted that we need enough computing power to simulate 86 billion neurons in realtime before computers can show any glimmer of intelligence.

> "It is similar to the risk of aliens attacking earth after detecting its TV and radio transmissions."

We have high confidence that there's no quicker way to get here than the speed of light. We know that space travel is vastly complex and expensive, so the kind of reasons humans went to war in the past (land, resources) do not apply to aliens - any species capable of making interplanetary warships can synthesise water, mine asteroids, build Dyson swarms, cheaper and quicker than coming here to take them from Earth. Even if they did want to destroy us, it wouldn't be Independence Day and Will Smith dogfighting, the aliens could piledrive Earth with one spaceship moving at interstellar speeds and bam, extinction level event we'd never see coming or have a chance to react to. When the meteorite crash extinguished the dinosaurs, the impact was like a megaton nuke every six kilometers and lead to hours of sustained inferno as all the displaced rock rained back to Earth, being on the other side of the Earth didn't protect dinosaurs from being cooked.[1]

A thing with power which is also untrained, untamed, clumsy, unaware, fundamentally alien without even the shared mammal / living creature history, doesn't have to choose to attack us it can potentially end our rather fragile lives with its initial thrashing about.

> "given how remote the associated risks are. The concerns about AI risks are like a drunk driver going 90 mph on a country road and suddenly deciding to stop saying "goddamn" just in case Jesus returns and punishes those who invoked God's name in vain in violation of the Third Commandment."

GPT1 (2018), GPT2 (2019), GPT3 (2020), GPT4 (2023), LLaMA (Feb 2023), TogetherAI released LLaMA training set (April 2023), LLaMA-2 (July 2023), llama.cpp (July 2023) now with >1500 releases since then, Mistral AI (April 2023), Anthropic Claude 3 (March 2024), Bard, LaMDA, Bing chat, BloombergAI, Bart, Gemma, Gemini, Grok, Sora, Falcon, https://llmmodels.org/

I don't think a present-day LLM is going to be the superintelligent AI, but in the last ~5-10 years we've poured billions of dollars, tens of thousands of the world's smartest information processing people, the resources of the world's biggest companies, the greed and investment resources of the world's VCs, the open collaborative spread of the internet, and added the heat and chaos of hype and FOMO and nationalist competition and dangled results (like SoRA) that machines have never been able to do before. If this doesn't ignite it, maybe it cannot be ignited. But if it can be ignited, we're trying hard. Could it be as far away as 2200? 2100? 2050? Could it be as near as 2040? 2035? 2030? 2026?

To just handwave this away as "the risks are remote" isn't convincing. The risk is on our doorstep, Aladdin's cave is open, the lamp is found and people are shoving pipecleaners down the spout, cupping their hands over it and calling "halloooooo, is anyone in theeeereeee? Genieeeeee?". The lockpickers are at Pandora's box, and Pandora is stepping away and looking nervous. Louis Slotin is showing off holding the two halves of the AI Demon Core apart with a screwdriver we're the bystanders saying to each other "this can't be dangerous there are no aliens in the room, lol". The future is here, it's just not evenly distributed yet - well if the AI wakes up[2] we'll have a sixteenth of a second to understand and respond before light has distributed its influence both ways around the Earth and met on the other side. Our main hope is that there is no Genie and that Pandora's box is past its use-by date and the contents are dust. Because we sure aren't building a suitable containment chamber, cautiously scanning it, and standing well back.

[1] https://www.smithsonianmag.com/science-nature/what-happened-...

[2] https://scifi.stackexchange.com/a/28116 - Dial F for Frankenstein by Arthur C. Clarke ~1965

[dkjaudyeqooe]

OK, so how does AI get its hands on nuclear weapons?

[jodrellblank]

If your thinking is along the lines “if jodrellblank can’t convince me that a superintelligence is plausibly dangerous, off the top of their head, before my attention span wanders, then AI can’t be dangerous” that’s a really weak plan for protecting humanity.

Say, pops up on the US or Kremlin computers and blackmails or bribes or threatens someone to launch the nukes? Say, places orders with a lab to build some robots which then become the AI’s body and it builds nukes itself. Say, placed orders for custom biological things using stolen money which turn out to be extremely virulent and then it doesn’t need nukes. Say it finds out an effect to worsen global climate change and uses it to make earth uninhabitable on a short timescale, and doesn’t need nukes.

[richardw]

Social engineering, for a start. Everyone with a secret will be a target.

Half the world will follow stupid leaders. An AI would run rings around your average voter. I'd be surprised if it can't run rings around basically every voter given time.

We're also creating various kinds of robots, so the thing could just plug itself into the network it wants.

That's 5 minutes. An AI would be smarter than me and have a lot more time every second to think up better ways, and A/B test them on millions of people.

All it needs is one guy thinking "Ah, what's the worst that could happen? It promised me a million dollars."

[throwup238]

Some contractor creates an internal webhook for testing the silo door that later gets accidentally hooked up to the full launch sequence and - during a billion dollar DoD mainframe upgrade project (to finally get away from COBOL) - gets exposed to the internet. MAD doctrine does the rest.

That is how the Anthropocene ends and the age of the machines begins. So say we all.

[Art9681]

This is the kind of nonsense that happens at startups, not government agencies with the capability to destroy cities.

[NateEag]

It sounds like you may not have read the "broken arrow" incidents Wikipedia page:

https://en.m.wikipedia.org/wiki/List_of_military_nuclear_acc...

There are a number of hair-raising stories tucked away in there.

Paired with knowledge of things like Stuxnet (https://en.m.wikipedia.org/wiki/Stuxnet), the broken arrow list fills with worry for what might happen with nuclear weapons systems in our modern, hyper-connected world.

[pksebben]

please see https://en.wikipedia.org/wiki/Nuclear_close_calls

[blibble]

if trump can figure it out an AI probably could

[dkjaudyeqooe]

You don't even need to be Putin. If you cook up the right chemicals, or breed the right virus, you can kill many thousands of people, maybe millions.

[tavavex]

Right? Humans are more than capable of severely harming civilization at any point, but for some reason this AI narrative is so much more compelling than the boring old issues we're used to. I still don't get the reasoning - we have pretty good text and image generators, yes - but the article jumps the ship to a whole different universe, one where "AI" is an actual entity that has needs and that outpaces humans in all possible facets. Yet, despite the wide gap between what we have and what they're talking about, this level of AI is treated as if it's not just feasible, but basically already here.

The article constantly refers to a strange niche sub-community of pro-AI people, and extrapolates it to say that almost anyone who backs new technology is a hyper-capitalist libertarian who just wants to see the world burn for the sake of money. I feel that the opposite ideology is also far from pure - with this immense reaction to generative AI, it almost feels like big companies are capitalizing on fear to promote regulations that shut out anyone who's not a big company that publishes fancy charts about "risks of catastrophe".