[d--b]

AI is not going to become self aware and destroy the world.

AI is going to cause something like the industrial revolution of the 19th century: massive changes in who is rich, massive changes in the labor market, massive changes in how people make war, etc.

It’s already started really.

What worries me most is that as long as society is capitalist, AI will be used to optimize for self-enrichment, likely causing an even greater concentration of capital than what we have today.

I wouldn’t be surprised that the outcome is a new kind of aristocracy, where society is divided between those who have access to Ai and those who don’t.

And that I don’t think falls into the “Ai safety” field. Especially since OpenAi is Vc-backed

[siboehm]

You can be very worried about the medium-term dangers of AGI even if you believed (which I don't) that consciousness could never arise in a computer system. I think it can be a useful metaphor to compare AGI to nuclear weapons. Currently we're trying to figure out how to make the nuclear bomb not go off spontaneously, and how to steer the rocket. (One big problem w/ the metaphor is that AGI will be very beneficial once we do figure out how to control it, which is harder to argue with nuclear weapons).

Most of these AGI doom-scenarios require no self-awareness at all. AGI is just an insanely powerful tool that we currently wouldn't know how to direct, control or stop if we actually had access to it.

[csmpltn]

> "Most of these AGI doom-scenarios require no self-awareness at all. AGI is just an insanely powerful tool that we currently wouldn't know how to direct, control or stop if we actually had access to it."

You're talking about "doomsday scenarios". Can you actually provide a few concrete examples?

[marvin]

Over the course of years, we figure out how to create AI systems that are more and more useful, to the point where they can be run autonomously and with very little supervision produce economic output that eclipses that of the most capable humans in the world. With generality, this obviously includes the ability to maintain and engineer similar systems, so human supervision of the systems themselves can become redundant.

This technology is obviously so economically powerful that incentives ensure it's very widely deployed, and very vigorously engineered for further capabilities.

The problem is that we don't yet understand how to control a system like this to ensure that it always does things humans want, and that it never does something humans absolutely don't want. This is the crux of the issue.

Perverse instantiation of AI systems was accidentally demonstrated in the lab decades ago, so an existence proof of such potential for accident already exists. Some mathematical function is used to decide what the AI will do, but the AI ends up maximizing this function in a way that its creators hadn't intended. There is a multitude of problems regarding this that we haven't made much progress on yet, and the level of capabilities and control of these systems appear to be unrelated.

A catastrophic accident with such a system could e.g. be that it optimizes for an instrumental goal, such as survival or access to raw materials or energy, and turns out to have an ultimate interpretation of its goal that does not take human wishes into account.

That's a nice way of saying that we have created a self-sustaining and self-propagating life-form more powerful than we are, which is now competing with us. It may perfectly well understand what humans want, but it turns out to want something different -- initially guided by some human objective, but ultimately different enough that it's a moot point. Maybe creating really good immersive games, figuring out the laws of physics or whatever. The details don't matter.

The result would at best be that we now have the agency of a tribe of gorillas living next to a human plantation development, and at worst that we have the agency analogous to that of a toxic mold infection in a million-dollar home. Regardless, such a catastrophe would permanently put an end to what humans wish to do in the world.

[tome]

> they can be run autonomously and with very little supervision produce economic output that eclipses that of the most capable humans in the world.

What’s the evidence for this?

> Perverse instantiation of AI systems was accidentally demonstrated in the lab decades ago

What are you referring to?

[nopinsight]

By the time you find such evidence, it could already be close to game over for humanity. It’s important to get this right before that.

We already have significant warnings. See for yourself if latest models like Imagen, Gato, Chinchilla have economic values and can potentially cause harm.

[marvin]

Historical examples of perverse instantiation are everywhere: Evolutionary agents learning to live off a diet of their own children, machine learning algorithms attempting to learn gripping a ball cheating the system by performing ball-less movements that the camera erroneously classifies as successful, an evolutionary algorithm to optimize the number of circuit elements in a timer creating a timer circuit by picking up an external radio signal unrelated to the task and so on. Some examples are summarized here: https://www.wired.com/story/when-bots-teach-themselves-to-ch...

GP wanted a concrete example of a doomsday scenario of failed AI alignment, so in that context extrapolating to a plausible future of advanced AI agents should suffice. If you need a double-blind peer reviewed study to consider the possibility that intelligent agents more capable than humans could exist in physical reality, I don't think you're in the target audience for the discussion. A little bit of philosophical affinity beyond the status quo is table stakes.

[davesque]

Agree with basically all of your points. I have huge concerns that the humanity of the future will basically split into two different species: the technocracy and the underlings. Sounds like science fiction but it honestly feels like we're headed in that direction. Even today, the privilege afforded by a life in technology and among technologists seems to set a person apart from the rest of the world to such an extent that they almost forget it exists. It feels like such a technocracy would have no moral right to exist. It can't really just be survival of the fittest, can it? I'll just keep believing (pretending?) that the answer is no.

[sgillen]

>> Even today, the privilege afforded by a life in technology and among technologists seems to set a person apart from the rest of the world to such an extent that they almost forget it exists.

I agree on your second point, but those in medicine, finance, or law enjoy similar salaries and quality of life to those in tech. Furthermore to really set yourself apart and join the global super rich you can’t really do that by selling your labor no matter your field.

[d--b]

Medicine, finance and law are three disciplines that are being heavily threatened by AI.

[sgillen]

You can add many tech disciplines including large swathes of programming to the list of things being threatened by AI.

[atmosx]

To have access to the forefront of AI you have to be super-rich. I don’t see how AI will change that, if anything will make it harder to change by giving yet another advantage to those who already have plenty.

[zzzzzzzza]

uhm, i spend less than 40 cents a day on avg on gpt3 queries...

a bit more accessible than like a hackerspace membership or building a factory or something

[omnicognate]

You don't have access to the forefront of AI. You have the ability to give money to those that do so you can use what they've made.

To have access to the forefront of AI means being able to make, own and profit from things like GPT-3, and it requires access to vast computational and data resources.

[sva_]

I doubt GPT-3 is profitable as of now.

[omnicognate]

That's not the point. Whether OpenAI is (yet) profiting from its models doesn't change the fact that GP lacks the resources to do what OpenAI is doing.

[csee]

  "AI is not going to ... destroy the world."

Bare assertion fallacy? This question is hotly debated and I don't believe it can be so easily dismissed like that. It is not obvious that aligning something much smarter than us will be a piece of cake.

[d--b]

Should I really add “in my opinion” to all the sentences I write? We are a smart bunch here. We can figure out when statements lack nuance in order to provoke some reaction.

We’re talking about the future here and a fairly complex one at that. So obviously I don’t know more than the next guy.

[tlringer]

It's a really absurd opinion that AI will destroy the world, and one that does not deserve serious consideration in any research community. It's only in strange Rationalist corners and the companies in Silicon Valley that echo those corners that this is considered at all "hotly debated."

[csee]

Why do you think it's absurd? If we do eventually create an AGI that is significantly smarter than us in most domains, why is it that we should expect to be able to keep it under control and doing what we want it to?