[jeremyjh]

I agree that evolution could not produce a rational agent who would still reliably respond to lower level imperatives (such as pain, hunger, lust) without consciousness and feeling. The primitive parts of the brain have to be able to override the higher functions to ensure survival and reproduction. But an LLM isn't evolved in this way; its fitted to a functional output. It is entirely possible there will never be anyone home. I sure hope there isn't, because at the scale we're using them it would be a moral catastrophe.