[houzertuch]

I don’t understand how there can be no comments regarding the fatal flaw of AGI which is that it will completely ruin the economics of the world. The world is the way it is now because humans are the only source of intelligent signal processing. That’s the only reason why humans enjoy the limited rights and privileges that they do. That’s the only reason why life has gotten better and better with advancing medicine and so on. This is a fundamental principle that cannot be escaped. It doesn’t matter how you slice it. But people defer everything to “ubi will work out somehow” or “nah humans will never be replaced.” Bringing god-like super-intelligent beings online is a fundamentally stupid thing to do. And preventing their development, relative to how disastrous their development would be, is very easy.

I have made many predictions here on HN and they all have outlined that cloud computing would be the substrate from which AGI will spring. Now we see this announcement. There is a reason why OpenAI is making a deal like this with a very large cloud compute vendor: it’s because I’m right. And that means I’m probably also right in saying that we can stop this if we want to. You can’t just build a computer in your bad yard. And the internet is very fragile. Some simple regulation and global awareness and initiative could control what comes out of fabs and shut down the infrastructure necessary for cloud computing. It would be very easy relative to the size of the problem.

[richardwhiuk]

Is your alternative "Don't invent general AI?"?

If so that seems unbelievably naive. Things generally can't be un-invented, and it's unimaginably hard to prevent people inventing things, especially with such a large economic upside for inventing it.

[houzertuch]

Prove me wrong if it is so obvious.

As the people at OpenAI have rightly said, AGI is a compute-gated problem. It is a problem that can only be solved with very, very large amounts of compute.

The world has some total amount of computing power in terms of silicon based computation. For AGI to happen, there are two requirements: that this total be equal or greater than some theoretical threshold value for AGI and that the computing power is consolidated. So in layman’s terms, you have to have a lot of computers and they have to be connected in such a way as to efficiently share their compute. AGI will never come about if every individual computer were used to do research by separate entities but if all of those computers were connected into a single virtual computer, AGI might be discovered with them.

So clearly in order to prevent AGI, the best thing to do would be to address these two aspects. Prevent the total computing power of the world from growing and prevent computers from forming virtual meta-computers. Both of these tasks are in principle extremely easy.

Chip fabs are huge and expensive. Nobody is fabricating chips in their garage. This is just a hard fact. There aren’t that many fabs in the world and they are all highly susceptible to regulation. This isn’t prohibition of alcohol so please don’t confuse yourself. Nobody will be brewing chips in their cellar.

Let’s imagine that you could not regulate cloud computing. Let’s say the only way to prevent computers from offering their compute on a virtual market was to shut down the internet. This by default is the hardest way to solve the second aspect of the AGI problem and even it is very easy. This is because the internet is a large fragile collection of infrastructure that depends heavily on government cooperation. Nobody is going to string a fiber backbone for black market internet. ISPs cannot exist without regulatory approval.

If there were political awareness and motivation, and it was a global phenomenon, yes, it would be extremely easy to do what I’ve described. And since AGI is to the detriment of literally all people, it is not a far fetched scenario. And unlike alcohol in the United States, bootlegging would not be a problem. People in the USA think that banning anything whatsoever doesn’t work. It’s just fuzzy thinking, I can assure you.

[richardwhiuk]

Nothing in that sounds easy.

If there are N governments in the world, and they all agree to not regulate to not create general AI, then it's strongly in all of their interests to betray the others, create general AI, and capture the economic growth.

Even if general AI is impossible, it's in their interests to develop huge computing capacity, because that's demonstrably economically useful.

You are hypothesizing that's it's easy to get 7 billion people to all agree to co-operate in a game of prisoner's dilemma, when if a small fraction of them choose to betray, they have the potential to capture massive value.

And you want to do this under the premise that AGI might be a problem.

[houzertuch]

Like I have said countless times, it’s easy compared to what we get in return. It’s easy to understand in principle. It doesn’t require sophisticated mathematics.

So you think that we should let all countries have whatever weapons they want under your logic. They will develop nukes and chemical weapons regardless of any international agreement that is established, so why even try? The obvious answer is to advance our own nuke technology as fast as possible so that we, the good guys, will lead where the arms race goes.

And AGI is a far greater existential threat than nuclear weapons. It is a greater existential threat than anything else, including global warming. It’s the biggest Pandora’s box in history. The idea of controlling or guiding its impact by having “the good guys” develop AGI first is the precipice of naivety. We lose nothing by trying to stop it. And we stand to gain more than we have ever gained from any coordinated effort. How easy or hard it might be is irrelevant, although it is much easier than basically anyone appreciates.

[richardwhiuk]

Nuclear weapons provably kill people. AGI doesn't. Be careful about your hyperbole.

[epiphanitus]

>>The world is the way it is now because humans are the only source of intelligent signal processing. That’s the only reason why humans enjoy the limited rights and privileges that they do. That’s the only reason why life has gotten better and better with advancing medicine and so on.

That's actually an interesting point I haven't thought of before. I disagree with you though. You could have made the same argument during the industrial revolution, but capitalism, democracy and civil liberties are still around.

Also, you're assuming that human symbiosis with AGI is not possible. Nature is full of examples of organisms that are in symbiotic relationships. Of course, nature is also full of organisms that kill and eat and parasitise other organisms.

But the advantage that we humans have is that we are actually creating AIs/AGIs. And since we are creating it, we have influence over what it becomes, whereas, your average clownfish really doesn't have a say over what form his anenome takes.

And if that is the case, wouldn't you want free societies to have expertise in these technologies and to be fine tuning them so that they work for the common benefit?

Why should anyone believe that we can stop authoritarian govts from developing AGI when we can't even stop them from enriching uranium, where the capital costs are much higher?

For better or for worse, it is almost always the case that the difference between societies that hold economic and political power and those that don't is technology. I live in the US, I enjoy my civil liberties, and I would much rather we master these technologies ourselves and have the security that wealth and power provides us. Not saying democracy or capitalism are perfect, but they beat all of the alternatives.

Edit: added another thought

[houzertuch]

Industrial revolution: has nothing to do with this. Some forms of human labor were automated. The set of all human abilities was not threatened. AGI is fundamentally different because AGI will automate everything a human can do. Increasingly important distinction. And not something I would overlook in the hundreds of hours of rumination I’ve put into it.

Can’t stop countries from doing it: yes, we can. A tin can dictatorship can’t do it. And besides, it’s a problem that depends a lot on academic work being published and shared among researchers. It’s a collaborative effort largely. Some rogue state trying to do this while avoiding inspections and inquiries by larger nations, and with no support from reading publicly published papers from other researchers, it would be tough. But with a large majority of anti-ai countries we could do a lot to prevent even covert attempts.