[throwaway173738]

I think consciousness is going to turn out to be very challenging to define rigorously enough that we can test for its presence or absence. Emergent or not, the question is how do you determine when it has emerged? Is it a quantity or an attribute? Discrete or continuous? Does it have a finite or infinite range?

We can all agree on what color something is, but we can’t describe the color a priori, only by example. I think consciousness may be a similar phenomenon and the only test is by shared experience. If so then we are in deep trouble because we will not be able to anticipate when a system becomes conscious.

[darkwater]

I like the color example, but colors can be reproduced at will by trial and error and by observing the results (and collectively compare them to our shared experience).

Why cannot this be applied to consciousness as well? I mean, it's surely much more difficult to do compared to colors but... impossible?

[adaml_623]

You and previous comment seem to agree that trial and error/shared experience can determine if consciousness has emerged. And this might/will be challenging.

Previous comment used the word "anticipate" and I think they mean that we won't know in advance before we run the trial and error process.

When they say "deep trouble" I assume they mean because creating a non-friendly conscious AI might pose an existential risk for humanity.

However there is also the ethical issue of creating a consciousness and then destroying/murdering it.

[throwaway173738]

Basically all of that, yes.