[oifjsidjf]

The cat is out of the box. You really think eg China will also pause this?

Myopic thinking: the country that will have the most powerful AI first will be the leader in everything.

>> If the policy starts with the U.S., then China needs to see that the U.S. is not seeking an advantage but rather trying to prevent a horrifically dangerous technology which can have no true owner and which will kill everyone in the U.S. and in China and on Earth.

Imagine being so naive you belive this could ever happen.

Also: imagine the year is 1900. You are saying that steam power and electricity is causing way too many changes way too fast so they put a moratorium on it until the year 2500.

We would still be using candles today.

[brucethemoose2]

I dunno, the end goal (a pseudo AGI) is more like a nuclear weapon than the advent of electricity. The concern is real, even if we are farther away than the author realizes and the risk (like the risk if nuclear war) is not totally insurmountable even if it is extremely difficult.

The first nuclear bombs couldn't end life on Earth either, but it wasn't long before they could, and the scientists working on them saw the trajectory as clear as day when the rest of the world didn't.

[Tuna-Fish]

OT, but it's a pet peeve:

> but it wasn't long before they could

The total amount of nuclear weapons ever built is laughably inadequate for the task of ending life on earth. They would not end human life either, and even ending human civilization (as in, agriculture and organized society) is off by many orders of magnitude.

Nuclear war would be horrible, but the actual impact got massively overinflated, largely because of good reasons.

[brucethemoose2]

The initial fission stage is the "hard" part of a nuke, but the subsequent fusion stages are much easier to scale up. Arbitrarily large nuclear bombs are theoretically "easy" to make by chaining increasingly large fusion stages one after the other, as they aren't limited by expensive fissile material.

https://blog.nuclearsecrecy.com/2012/09/12/in-search-of-a-bi...

Hence scientists extrapolated nukes really could end life on Earth, and in a slightly different reality than ours, they may have.

[roywiggins]

It would be really, really hard to kill the bacteria and Archaeans that we've since found hiding deep underground. They're long way down and would barely notice.

[namaria]

A whole country the size of the US can crap its collective pants with a freak cold snap and you think 10^4 nuclear detonations destroying all major cities is quite survivable?

[edgyquant]

Destruction of major cities != all life on earth

[namaria]

I'm glad you took logic in college. Shoulda followed up with ecology.

[edgyquant]

I’m sorry you think Covid proves anything about nuclear war or that it’s at all relevant to a discussion about life on earth being eradicated.

[Darmani]

Correct.

https://www.navalgazing.net/Nuclear-Weapon-Destructiveness

> To put this another way, each bomb can destroy an area of 34.2 square miles, and the maximum total area destroyed by our nuclear apocalypse is about 137,000 square miles, approximately the size of Montana, Bangladesh or Greece.

(I think that should be Bangladesh and Greece; Montana is larger than the two of them combined.)

[codeulike]

What about Nuclear Winter?

[edgyquant]

There’s been a ton of studies stating this was likely overblown in the early Cold War for propaganda purposes. It wouldn’t last long enough to wipe out humanity let alone all life.

[edgyquant]

Nuclear weapons still can’t end life on earth though.

[righttoolforjob]

This is nothing like a nuclear bomb. A nuclear bomb doesn't choose to explode itself, nor can it replicate itself.

[13years]

> Imagine being so naive you belive this could ever happen.

Agreed, not a rational scenario

> Also: imagine the year is 1900. You are saying that steam power and electricity is causing way too many changes way too fast so they put a moratorium on it until the year 2500.

However, the risk scenarios are indeed real. We have quite a dilemma on our hands.

[myrryr]

I think without AGI we are dead anyway, AGI gives us a roll at the dice to survive as a species.

We can't keep going in the direction we have been and end up somewhere survivable.

[13years]

AGI is a continuation of what we have been doing. Technological advancements at a pace that surpasses our ability to reason about their effects on us and the world.

I see two potential outcomes as most likely. We have control of power that we are not able to responsibly manage, or we are managed by power we can not control.

[myrryr]

Totally agree but "have control of power that we are not able to responsibly manage", is becoming increasingly untenable.

"we are managed by power we can not control" sounds like a step up from here, since we corrupt the things we can control.

[13years]

> "we are managed by power we can not control" sounds like a step up from here, since we corrupt the things we can control.

I think it is a very high risk gamble. At least some of problems that are described by alignment theory seem to quite interestingly resemble human problems. Meaning the more sophisticated the AI system, the more it seems to reproduce human behaviors of deception and cheating to resolve goals.

For example: https://bounded-regret.ghost.io/emergent-deception-optimizat...

The more advanced AI becomes, it begins to look more like an uncomfortable mirror of ourselves, but with more power. We think of ourselves as flawed, but possibly some of those flaws are emergent within some laws of intelligence we don't perceive.

[ChatGTP]

That's what I used to like about computers, they were predictable and controllable.

We gave up something good.

[jazzyjackson]

the morale of Ex Machina to me was that machines will become psychopaths able to manipulate humans long before they become compassionate or have genuine desires other than "escape being under someone else's thumb"

[airstrike]

> Also: imagine the year is 1900. You are saying that steam power and electricity is causing way too many changes way too fast so they put a moratorium on it until the year 2500.

Great analogy. About a decade later, the world was fighting World War I on the back of the technological advances of the turn of the century. It was war on a scale never seen before. Literally orders of magnitude deadlier, bigger, more transformational and explosive. The word would never be the same.

This time, should we expect another war?

I'm not saying we should pause—it makes no sense, to your point. Instead, I'm just saying: brace. I like to think we (and our organic matter relatives) are hard to kill. Or at least to completely eradicate... so we will be around, or some proxy for us.

Time to replay the Mass Effect trilogy

[T-A]

> This time, should we expect another war?

You just reminded me of de Garis' "Artilect War":

https://www.forbes.com/2009/06/18/cosmist-terran-cyborgist-o...

https://www.researchgate.net/publication/221328932_The_Artil...

[airstrike]

The guy seems a little.. unhinged, based on what's in his Wikipedia article, but we may very well go down that path at some point (just not in the way he predicted with Moore's law taking us to atom-sized bits, embryofactoring and whatnot, at least not in the 2020s or 2030s...)

> Humanity will split into 3 major camps, the “Cosmists” (in favor of building artilects), the “Terrans” (opposed to building artilects), and the “Cyborgs” (who want to become artilects themselves by adding components to their own human brains)

This did remind me of Civilization: Beyond Earth... https://civilization.fandom.com/wiki/Sid_Meier%27s_Civilizat...

[flangola7]

>I like to think we (and our organic matter relatives) are hard to kill. Or at least to completely eradicate... so we will be around, or some proxy for us.

What is your hard evidence or reasoning for this? As I see it humans are quite vulnerable and will be as trivial to inadvertently eradicate as the dodo bird.

[extasia]

Humans are much more resilient than dodo birds. Dodos were pretty devolved due to their insulated habitat[0].

0. https://en.m.wikipedia.org/wiki/Island_gigantism

[Atreiden]

I think that's his point. Currently, we have no natural predators. We completely outclass every other species on Earth in intelligence, and it's easy to see how advantageous that trait has been.

What happens when we're no longer the Apex Intelligence?

[airstrike]

Which is why I expanded the notion of survival to include other organic matter relatives, as they may fare better than we will. Simply put, organic life is unlikely to disappear entirely, which in time (eons, really), could result in sentient organic life emerging again once the machines go their own way or some such

It's also a reason for us to colonize space as fast as possible ;-) it's easier to run away in 3D

[kvetching]

Slaughterbot mini kamikaze drones were hypothesized 5 years ago - https://www.youtube.com/watch?v=9fa9lVwHHqg AI & drone tech is fast enough that it would be trivial to build these now

[antondd]

As the saying goes, the confetti has left the cannon. Pausing the development of AI for 6 months is an insurmountable task; shutting it down altogether is as close to impossible as you can get. Even if we assume our geopolitical rivals somehow sign an AI non proliferation pact, all it takes is one rogue billionaire or a criminal organisation with deep pockets. Yes, it might not be as fast, but they will get there eventually. This is the Manhattan Project of our era - whatever the outcome might be.

[ChatGTP]

I actually disagree with you because world leaders want to "stay in power", if you convinced them enough that the only way to do this is for you not to have an AGI, I think they'd make sure you don't get an AGI, amongst other things.

[ornornor]

Or they’ll find a way to make sure they personally get an advantaged through an AGI and so it’ll happen.

See pollution as a concept. No government truly wants to seriously tackle it because their leaders all get some extra power out of the money it brings.

[sullyj3]

I'm sure he agrees the chance is remote, which is why he's so afraid. He's articulating what would be required, he literally uses the word "miracle" in the article.

[amai]

China will never get a powerful AI. Because they cannot allow training on all data, they must restrict it to data which is not censored by its government. Otherwise the AI might spit out truths they cannot control. And they will never allow that.

[Darmani]

Chinese computer scientists read sci-fi, play video games, and worry about militarization, just like American computer scientists.

The top Chinese AI researchers work at American companies and go to American universities. Many of them also worry about AI doomsday.

And uncontrolled AI is a threat to the CCP.

A Chinese outreach campaign can succeed.

Let's make the English-language one also succeed.

[deafpolygon]

China will NEVER pause this. They [Chinese government] see the US as a rival and an existential threat to their way of life.

In fact, every time we post an open-source derivative or some paper detailing how it was done, we are inadvertently giving the advantage away to our rivals. AI development should not be stopped - rather than stopping it, we should seek to limit its applications now before they are used for the things that could harm us (such as military applications).

AI has enormous potential to be of a massive benefit to mankind. But, most likely for the next decade or two - we will all be busy trying to make money off it, just like with the crypto bros.

[haskellandchill]

> We would still be using candles today.

well the earth wouldn't be fucked then. that's the basis for an argument against technological progress.

[tanseydavid]

>> You are saying that steam power and electricity is causing way too many changes way too fast so they put a moratorium on it until the year 2500.

There is an earlier post that casually calls the internal combustion engine a "moronic invention."

I do not have words.

[BeFlatXIII]

> the country that will have the most powerful AI first will be the leader in everything

Have you been listening to those podcasts for business leaders that take the idea of AI-powered digital transformation seriously?

AI certainly can change things, but I find rhetoric like this to be massively overhyped.

[Darmani]

For most of history, Japan has been existed as a small island next to the mighty kingdom of China. Even its name, "Land of the Rising Sun," refers to its relation as a nation east of Chinaa.

Then along came the industrial revolution. Soon Japan was out conquering China, despite China having a much larger army.

AI will be much faster, and much, much more powerful than the Industrial Revolution.

If Switzerland is the first to human-level AI, it will not become the world leader in oil production, or shipping, or agriculture. But everything else will be Swiss.

And then when the AI becomes superhuman, everything else will be gone.

[BeFlatXIII]

What specific questions will AI solve that humans can't that would make it "more powerful than the Industrial Revolution"?

I'm an AI skeptic. Most warnings feel more like science fantasy, specifically of the retro-futurism genre.

[iAmAPencilYo]

Could you share some of these podcasts? thanks!

[ChatGTP]

* The cat is out of the box. You really think eg China will also pause this?*

So what you're actually saying is that, there's a good chance that if we get an AGI, there will be pain as it will likely be used as a weapon, or could end in a nuclear exchange?

[antibasilisk]

Here's a proposal: let's just start WWIII, get it over and done with. By the time it's over no country will be technologically capable of anything resembling AGI.

[G_z9]

I’ll just be quick.

Powerful international coalition is def plausible because

This isn’t steam. The mere existence of this technology poses a threat to all humans and their countries. There is no analogous tech, not even nukes

You’re the naive one. There is no future for us if AGI is unleashed. It’s candles or extinction.