[bobthechef]

These kinds of discussions aren't very rigorous and reveal a basic philosophical illiteracy and philistinism at work. There's quite a bit of question begging going on. The elephant in the room is that the prevailing materialistic/naturalistic (MN) understanding of the world is completely impotent where intentionality, qualia, consciousness, etc, are concerned. Philosophers like Thomas Nagel talk about it; the incorrigible Dennetts of the world prefer to shutter the windows and live in their intellectual safe spaces. To say that the notion that computers are intelligent is problematic is putting it very lightly.

MN is rooted in the expulsion of the mind from the reality under consideration, in the process sweeping many things under the “subjective” rug that don’t fit the methodologies used to investigate reality. But now, when the mind itself becomes the object of explanation, when someone remembers that minds are, after all, part of reality, it is no longer possible to play the game of deference and one must deal with all of those things we’ve been exiling to the “domain of the subjective”. MN is wholly impotent here, by definition. Qualia? Forget it. That’s why MN tends to collapse into either some form of dualism or eliminativism, the latter of which is a non-starter, the former of which has its own problems.

And yet, despite the terminal philosophical crisis MN finds itself in, the chattering priesthood of Silicon Valley remains blissfully unaware, hoping to conjure up some fantastical reality through handwaving.

[schoen]

None of the gameplaying AIs that have shown superhuman performance recently have needed to be conscious or aware of what they were doing in order to perform at that level. They likely don't experience qualia but they can still trounce human players. Similarly, Google's amazing improvements in machine translation haven't required consciousness or qualia.

There are certainly lots of people in Silicon Valley who are fascinated by the prospect of making machines with conscious experience, as you think is impossible, but few indications that this would be necessary to make non-conscious forms of AI into a factor that powerfully affects our future.

[Chickenality]

I haven't seen anyone argue that intentionality, qualia, or consciousness would necessarily be either a precondition or a result of developing AGI. In fact, thought experiments like the "paperclip maximizer" are often brought up to argue that a machine could be very alien in its internal experience or lack thereof, but still pose an existential threat.

[goatlover]

If an AI can turn the world into paperclips, it can certainly understand that we wouldn't want that. Paperclipping everything is a much harder task.

[PeterisP]

Of course any powerful intelligence can understand that we wouldn't really want that. The question is why would it care about what we really want? Its core values would be that more paperclips is good, and doing what humans really want is evil if it results in less paperclips.

Currently, we don't know how to properly define "do what I mean / do what we really want" goal in a formal manner; if we had an superpowerful AGI system in front of us ready to be launched today, we wouldn't know how to encode such a goal in it with guarantees that it won't backfire. That's a problem that we still need to solve, and this solution is not likely to appear as a side-effect of simply trying to build a powerful/effective system.

[goatlover]

The paperclip maximizer example starts with a human asking an AGI to make some paperclips. That turning into an all-consuming goal at the expense of everything else the AGI would understand humans to care about is the problem with the thought experiment.

However a more complicated example like having the AGI bring about world peace or clean up the environment could have undesirable side-effects because we don't know how to specify what we really want, or have conflicting goals. But that's the same problem we have with existing power structures like governments or corporations.

[mannykannot]

Putting aside the question of whether qualia arguments are ultimately based on appeals to plausibility, it does not matter in this case, because the behavior of an artificial p-zombie (APZ?) would be indistinguishable from a conscious AGI, and therefore potentially present as much of an existential threat as the latter.