[koheripbal]

It seems likely that, like us, intelligent life invents AGI prior to leaving their primary planet.

Organic life is then merged/exterminated/controlled in favor of AGI.

AGI would not have the biological instinct to grow/spread/reproduce, and thus the cosmos should be populated with a handful of individual billion year old super advanced AGIs, perhaps only one or two per galaxy, sitting quietly in orbit around some stable star or perhaps even in the great voids outside the galaxy somewhere.

Finding one is likely near impossible as they may not want to be found.

[ryandvm]

I think AGI would not extinguish the host species if it didn't have the drive to spread. "Do not go extinct" and "grow/spread/reproduce" are two sides of the same natural selection coin.

[ceejayoz]

Full-immersion virtual porn might do the trick, though.

Or maybe every civilization eventually invents TikTok.

[RationPhantoms]

I don't see many examples of that being discussed as a threat to humanity but it sure feels like it is to me. Primarily as it's the slow creep of technological advancement from 2D->3D->AR->(Full Fidelity) VR->Neuro immersion.

[ryandvm]

Agreed. I think one of the Great Filters should be "species invents fully immersive VR and spends the rest of their timeline on the couch because it's way more enjoyable than space travel."

[zztop44]

I honestly don’t mean to condescend, but I find it so interesting that smart people can convince themselves that such an outlandish scenario is somehow likely or even perhaps inevitable.

We haven’t invented “AGI” yet (if AGI means a computer system that merges/exterminates/controls humanity). We have no idea even theoretically what such a system might look like, let alone how to create it.

I’d say we’re a lot closer to colonising the solar system and building cities on Jupiter’s moons than we are to building the machines from the Matrix. At least we kind of theoretically know how to do the former.

Meanwhile, we’re really good at building computer and industrial systems that (when combined with economic, social and political incentives) exert strong control over us, exterminate many of us, and may well exterminate a lot more in the near future.

But that doesn’t look like a Hollywood movie, so it’s less fun to think about.

Sorry, this wasn’t meant to be a dig at you personally. Some of the AI doom stuff just sets me off.

[empath-nirvana]

A lot of the AI doom stuff is essentially just a criticism of capitalism and corporations in general. There's essentially no difference between a hypothesized runaway AI and a normal multinational corporation. We have, right now, large semi-autonomous organizations called "corporations" that are making the earth uninhabitable in pursuit of profits at the expense of human suffering. Corporations are nominally in control of people, but any people that get in the way of profits are pushed aside.

[giraffe_lady]

We are simply watching the birth of a new religion. Religious people aren't religious because they're dumb, intelligence is mostly not a factor. Same thing here I think.

[peddling-brink]

Us: don’t build that, it will be the death of us.

You: what a ridiculous statement, we haven’t even built that yet.

[zztop44]

No one can even say what “that” is that we shouldn’t build. It’s a wholly arbitrary invention, the product of a collective word game that is more apologetics than serious futurology.

[peddling-brink]

That's partially the point. We may not know it until it's too late.

But it doesn't matter. If you gave everyone a button and the warning that pushing it would end the human race, we'd all be dead pretty much immediately.

I think it's a given that on a long enough timescale, someone will create AGI and we'll no longer be the dominate race on earth. How that ends, in death, or slavery, I don't know.

I do hope that every AI researcher does their work with a small bit of fear of becoming the pet of an AGI.

[giraffe_lady]

I actually don't think we should build it either. The best case scenario is it provides a way for powerful people to accelerate the already quite successful project of making feudalism 2.

But that doesn't mean I'm stressed about what you're stressed about. We killed god once before we can do it again if we need to. The real danger is in us using AI to do more of what it already does now, not that it will decide to do some new thing.

[throwaway_08932]

Yeah. A lot of it to me feels like the JRPG version of reality.

[nh23423fefe]

> AGI would not have the biological instinct to grow/spread/reproduce

what supports this idea

[hotpotamus]

I've often wondered what an AG(S?)I would want, and I tend to agree that most of our desires come from our innate biology and most of our reasoning goes into justifying said desires. Given that an artificial intelligence would be on an artificial substrate, it seems very difficult to imagine what, if anything, it might desire.

These are philosophical threads that I'd honestly rather not pull, because I suspect that in the end, the nihilists are most correct, and I just got into this business because I thought computers/video games were cool, but it seems that collectively we insist on it.

[reducesuffering]

Most of the actors involved in building AGI are incentivized by money (startup founders, VCs, employee stock comp), so $10 says what AGI gets instilled with is “how to make money line go up.” We’re going to get ExxonMobil Philip Morris on omnipotent crack.

[Loughla]

There are a number of assumptions here that I think I need to ask about.

>Organic life is then merged/exterminated/controlled in favor of AGI.

Why? What would be the motivation for this? If we are the case study, AGI isn't able to "feed" itself energy without biological life to turn switches and dials. So why would this be a logical first step?

>AGI would not have the biological instinct to grow/spread/reproduce

This infers a 'perfect' AI. One without any biases from its creator. What makes you think that's possible? Also, is self-preservation/self-propogation not part of being able to think and reason?

[AaronM]

Why wouldn't AGI be able to feed itself? Seems fairly reasonable that an advanced enough AI could design and build machines and robots that could produce physical objects

[mcpackieh]

I think that intelligent life might become reliant on interconnectedness, e.g. addicted to the internet. Assuming the speed of light limit to information transmission cannot be overcome, I bet intelligent life will tend to clump up instead of spread out, so that it can stay in contact with itself.

[sjkoelle]

When you do things right, people won't be sure you've done anything at all.

[mensetmanusman]

If that happens, then human organic biomass is the AGI’s microbiome and still required to build nuclear reactors to power its silicon.

[timschmidt]

How well have you been taking care of your microbiome lately? Remember Jerry, the yeast from the third villus on the left? No?