[agronomov]

It seems that one danger is often overlooked, which is humans believing AI has consciousness once it becomes advanced enough. It will start with dying people migrating to cloud hoping for eternal life, and on the outside it would look like they continue to live in the metaverse forever. With time the metaverse would look better than real world in every possible way and people would start to voluntarily move to metaverse, and then we all start living in the matrix. Except that in reality nobody would be living; these would be just algorithms mimicking human life. They would continue doing what humans do in the exact same way, perhaps even building humanoid robots, because that's what humans would do, and sending them to distant stars, because that's what humans would do as well.

All would be great except that there would be no one to experience this, because there is no artificial consciousness. But there would be no one alive to ask that question.

[tim333]

> there is no artificial consciousness.

But can you be sure there won't be? I mean if you say is the patient conscious and the doctor says yes he's looking around and talking so you then say than only counts if the processing is done by meat rather than silicon?

[agronomov]

I can't be sure because I don't know where the subjective feelings are coming form. But I don't think there's any basis for thinking that an algorithm, however complex, might start experiencing things.

That's not the point though. The point is that there's really no way to tell. Imagine your friend's mind is uploaded to the cloud on their last life's second, and you ask them how they feel. They respond that they are feeling better than ever. They have all of the memories of your friend, they behave like your friend in every possible way. Heck, for simplicity imagine they also have an android body attached to them. For all you care that's just your old friend. The copy is so perfect that there is no way on earth to tell the difference.

However, that would just be an imitation. Could it have feelings? Maybe. I doubt so. We'll never know that. And that not knowing is the root of all evil

[oefnak]

Why do you think algorithms can't be conscious? (Unless you believe in souls, that is..)

[agronomov]

Because these algorithms just imitate specific output for a given input. However, they are so complex, that our brains are not able to distinguish their activity from a real consciousness in any way. If you believe that imitation process can have consciousness, then you might as well believe that a rock is conscious too.

I don't believe in souls, but I do believe in qualia, that is subjective, conscious experience, simply because I experience it (see the hard problem of consciousness on wiki). I do not a second believe that external imitation of any type of activity can generate qualia, because there is not continuity (if one algorithm has consciousness, then slightly less advanced should have it too, then you come to the point when a piece of metal is conscious too, see ship of theseus argument)

[danielbln]

Our own conscious continuuity is an illusion. We only really experience a single moment at a time, moment to moment. Anything else is memory. What happens if you attach memory to a sufficiently advanced algorithm? Qualia? Subjective experience? I don't know, but neither do you or anyone else for that matter.

[agronomov]

I am not talking about consciousness though, I couldn't care less about it! I am talking about the subjective feeling of existence, which is what you and I feel in every one of those disjoint moments. And the only reason I know that you feel it, is because I feel it and you're like me. If you sere made of silicon, I wouldn't know, and should not assume that you could

[goatlover]

What would it mean for an algorithm to experience color, sound, pain? There's no need to bring souls into it. That's a religious straw man. We know we do experience the world through sensory sensations, and those also make up our thoughts, memories and dreams. What we don't know is how our understanding of the physical world can possibly create those sensations, since we model the world as abstract mathematical patterns that don't experience sensation.

[agronomov]

I'm not talking about souls! What the heck is that anyways. Yes, I'm talking about the fact that our understanding of how physical systems work (neurons and all that) give us no understanding of why we need to experience things. We could have worked just fine without it (again, see the hard problem of consciousness). So I assert that AI would function perfectly just fine without subjective feeling as well, and we would't know it because there's no way to establish it. However, we would fail the mirror test: it looks like us, behaves like us, hence it feels like us, but it's just a quality imitation.

[oefnak]

I believe it is not possible to imitate a human without the imitation being so complex as to experience reality itself. Your argument is basically the Chinese Room experiment, in which my view is that the whole room is conscious.

[booleandilemma]

Until the AI humans figure out how to create life, and then the cycle repeats.

[agronomov]

Nice one

[imgabe]

The end goal of meditation, as I understand it, is to come to the conclusion that consciousness is an illusion anyway. So perhaps it wouldn’t matter.

[goatlover]

The self, not consciousness. Eastern philosophy is more likely to be idealistic than materialistic. The self as a separate entity that has stability over time is the illusion. The experience of change is not an illusion. Nor would whatever experiences come with enlightenment.

[agronomov]

These are exactly the kind of words that would end humanity

[bell-cot]

I don't think that scenario actually works. Reason: the superficial, "sounds like a person" AI's don't actually have what it takes to sustain a huge, high-tech society.

OTOH, letting the idle rich "move" to the matrix could be a good thing...

[agronomov]

Well, imagine advanced AI beyond imagination. It is so advanced that it mimicks all of the human activity in exactly the same way. By definition it would be able to do whatever we are doing (and probably better)

Unless you don't believe that such an AI is possible, in which case I wonder if you have tried GPT-4 and whether you remember what chat bots were like 5 years ago!

This of course would be AGI. It would still not be conscious though, but a lot of people would fall victim to it and believe that it is. And that's what will end us. We will not pass the mirror test (yep, this one https://en.wikipedia.org/wiki/Mirror_test)

[bell-cot]

In a Philosophy 307 class, your "advanced AI beyond imagination...mimic all of the human activity..." description begs some questions about souls, or other "special sparks of magic" which somehow distinguish such AI's from real humans. And exactly how those souls or sparks or whatever might pass Carl Sagan's Invisible Dragon Test.

But in the real world - those same descriptive words are written by a Marketing Dept. The actual engineers, who developed and built the AI, on an all-too-finite budget, know otherwise. And know that the PHB's already have plans to cut the production cost - by collecting data on which parts of the already-limited "mimic all human activity" are ignored by the AI's actual customers, and quietly nerfing or removing those parts.

[agronomov]

That special spark of magic is my feelings that I experience all the time. I'm not talking about souls -- amoeba probably also experiences things (just by analogy form us)

[FredPret]

Advocating casual class genocide. Nice.