In Wei Dai's "nightmare scenario", what of value is lost?
If "consciousness", whatever that is, is completely undetectable by external means and has no effect on human behavior, I think we can safely ignore it.
That seems like a misreading of the scenario. Consciousness certainly affects behavior, like making me say "I'm conscious". The question is whether the same behavior could be reproduced by a more efficient program that isn't conscious, and if yes, how do we make sure uploads are conscious.
If the same behavior (including debating consciousness in internet threads!) could be reproduced by a more efficient program that isn't conscious, how do you know you are conscious?
Come to think of it, these threads do get a bit repetitive - are we sure this hasn't already happened? ;)
Some of my internal experiences are hard to express in words, like how the color red looks. It seems overconfident to claim that such experiences don't exist or don't matter.
Consider a person who is completely committed to keeping some secret (e.g. that they saw a UFO). Their input-output behavior is the same as a person who doesn't know the secret, but the internal experiences are different. Do you think the AI in charge of optimizing uploads should be free to discard such differences?