[stared]

I like this essay, and have very often referred to it when someone talks about AGI. There is a common narrative bias to look at AGI as the Abrahamic God, if not explicitly, then just by saying that it is omniscient, omnipotent, immortal - and will judge us for our deeds.

It is tempting for anyone raised in the West, and immersed in Judeo-Christian culture. And for anyone, in general, as it offers an epic narration of a personal entity.

Yet, the reality might be messier - IMHO closer to biology than to a weird mixture of computer science and theology. There is no ultimate intelligence (see Karpathy’s starfish shapes), just a collection of adaptability, learning, generalization and self-reference. Also, even an extremely smart being (or process) can be fragile.

So, less God, more WAU from SOMA or the Ocean from Solaris.

[smaudet]

For all its talk of inoculation, this is a terribly written essay. They do not make a point, nor even arguments, instead, opting to ramble in hopes that you forget whatever it was you were thinking.

The issue is simple. Just like us (who are arguably complex, look at what we're building over here, this AI computer stuff!), entities have simple core needs (like food, water, power, etc.).

An infinitely smart AGI has the potential, nay, likely cause, to require infinite resources. We're already seeing the effect in the computing sector on e.g. chips, there's no reason to think this trend won't continue...

Lets circle back to the hydrogen argument, will we blow ourselves up. Real concern, abated by hard numbers. Different atmosphere, different concentrations, different pressure, different possible outcomes.

Today, we don't have those numbers. We don't have those calculations. I don't disagree with the point at the end "about how people can exploit other people, or through carelessness introduce immoral behavior into automated systems". These are issues, too. But saying there are other issues, don't worry about this big issue over here, is the absolute worse argument possible.

That's hand waving.

[kbrkbr]

From my point of view we have simply no idea what a infinitely smart AGI is and how to build it.

How would it make the combinatorial explosion in state space search go away, to pick one example?

And if it doesn't, is it then an infinitely smart AGI?

The concept seems to assume all problems humans struggle with can be solved. The halting problem is one witness that this is probably not true.

[fragmede]

If you're gonna look at it that way, then the halting problem is just a dressed up computer science version of the question "can God make a rock so heavy even he can't lift it?" Could an infinitely smart AGI come up with the answer to the unanswerable question?

It doesn't need to be infinitely smart to do a better job than the worst of humanity's blunders.

[kbrkbr]

But isn't on the other hand the current AGI problem posed similar to the question "Are you not afraid that genetic engineers grow babies with bigger and bigger brains?" We don't know if that won't break down somewhere. Looking at different examples it probably will. Scaling things infinitely is a pure math only concept it seems.

[smaudet]

Again, infinity is illustrative.

Put another way, you do not deny a proof by inference because it "leads to large numbers".

We don't need to speculate, we can see many, many examples today of more and less intelligent species, and also what happens to the less intelligent ones, even taking humans out of the equation.

I'd argue, even a machine intelligence that merely managed to be mildly smarter than us would be a threat. AGI merely has the potential to be infinitely smarter than us, but that's somewhat irrelevant given we might not even be smart enough to realize how much smarter they are (a cat will likely not appreciate the difference in intelligence between a dog, an elephant, or a dolphin, despite all those animals being generally smarter).

[goatlover]

Infinities aren't a physical reality. Resource are always limited, physics is limited at the Planck scale. You can only do so much compute in a finite volume of space, and there will only be so much energy available.

As for simple needs, humans also have complex ones around social interactions and the need for mental stimulation.

[smaudet]

Infinity is....subtle.

They are more characterized by how they grow and when they stop than by their "physical reality". Proof of this is in that different infinities exist - characterized precisely by how fast they grow, i.e., one "infinity" is "larger" than the other.

My point is, getting hung up on "infinity" as being unrealistic is not the point. It is the tool with which to understand how thing behave. The same as any calculus problem - you take the limit to the infinity to understand how the function behaves.

[JumpCrisscross]

> We're already seeing the effect in the computing sector on e.g. chips, there's no reason to think this trend won't continue...

That effect is underwritten by economic demand and moderated by economic costs. There are more reasons to expect the trend to asymptote than somehow turn into an infinite process.

[tim333]

I've only heard the God narrative from skeptics as in look at those idiots thinking they are building God. I think most people who believe in AGI arriving see it more as something like a chess computer but as well as beating us at chess it'll do other thinking too. A souped up chess computer isn't God.

[stared]

I have heard it mostly from the Less Wrong crowd.

I mean, all stories about religious dedication to "alignment", with doomsday vision if we do it wrong, and a vision of paradise if we do it correctly.

In particular, the concept of the Roko's Basilisk is some rehash of the Pascal's Wager.

[Jach]

Would you care to share any links? Specifically any talk of an AI-God "judging". If your entire perception is based on what you think you know about the Roko incident, may I suggest that you are ill-informed about it? It wasn't something the "LW crowd" endorsed or believed. Looks like I left some links to help someone else become informed several years ago: https://news.ycombinator.com/item?id=18982933

[stared]

For starter: https://www.lesswrong.com/w/paperclip-maximizer-1

[Jach]

I can see how talk of a paperclip maximizer, or any sort of AGI that can actually deliver on e.g. Drexler's nanotech or otherwise act in very powerful ways like, to quote, "colonizing the galaxies", pattern-matches to something roughly omnipotent by most of our standards. I can see how it roughly pattern-matches to something like "immortal", but this seems like the least of feats, much of our complex machinery with maintenance is effectively immortal already. Overall I don't see how this connects to what you originally wrote: "There is a common narrative bias to look at AGI as the Abrahamic God, if not explicitly, then just by saying that it is omniscient, omnipotent, immortal - and will judge us for our deeds." It is not omniscient, it only knows enough to develop the tech to enable paperclip maximization. It is not judging, either: "The AI does not hate you, nor does it love you, and you are made of atoms it can use for something else."

I should also note that the paperclip maximizer is not something the LW crowd believes should exist. Its primary function as an idea is to illustrate the orthogonality thesis: that goals and intelligence aren't dependent on each other. Its secondary function is to illustrate instrumental convergence.

[wamatt]

> In particular, the concept of the Roko's Basilisk is some rehash of the Pascal's Wager.

Oh boy couldn't agree more. The whole Basilisk drama really caused me to rethink the idea that a group of Rationalists were in practice, were in fact particularly rational. CF The Atheists vs someone that happens to be atheist. To their credit, they also acknowledged this somewhat and hence the CFAR crowd.

The word "rehash" also resonates. Like Google has a tendency to use PHD's to reinvent everything over and make up new words and terms for the same concepts.

Having spent a little too much time in the early 2010's at Bay Area LW meetups, the more fantastical LARP of fanfic, and religious mythology often felt a bit at odds. There was the aspect of a charismatic, autodidact leader obsessed with a certain J.K. Rowling IP and the kinky stuff..Don't get me wrong, I still have fond memories of this time overall :)

From my perspective, a core issue seemed to be no-one seemed to particularly motivated in defining what it meant to be rational, aside from some loose segmentation around instrumental vs epistemic rationality. (ie practice vs theory). And because of this it almost had a faith vibe to the scene. Like "trust me bro" this is "super high brow nerd stuff" and on your third helping of "The Sequences" all these formulas and shiny new words will all make sense what and it will be clear why we’re doing these meetups. (It totally wasn't anything to do with mental masturbation and high-iq crowd bonding and feeling good ;)

When I was into Christian apologetics (C.S. Lewis etc) as in my first year of CS & philosophy of science in college, there was a similar thing.. after a year of seeking out the scientific, and logical explanations for all the religious dogma I was indoctrinated into growing up. In the end, it pretty much reduced to "just have faith". This was after exhausting the "well you're not an expert on Christian theology, so you cant have a solid argument around the nature and existence of God because you need to study more" counter. This despite reading and studying the Bible at length.

For example, right now. Is it rational to be typing this up on HN, when I have other more important goals to do? OTOH reminiscing on the past and connecting with a single serving online friend OP, gives a bit of a dopamine hit. And maybe sharing resonates with others and increases happiness in the world? (or not if anyone still LW reads this and maybe feels a different type of way)

So that’s community right and good (in the sense it's aligning with goals)? But then its driven by emotions, so that kinda is not supposed to be rational. Is it rational to observe one's own mental states and take action? Turtles all the way down!