[qnleigh]

A lot of the comments in this thread point out that LLMs could be very good at tricking us into thinking that they are conscious because they operate on language; our brains are primed to empathize with what they say and imagine a being operating behind them. This is a really good point.

I'll even take it a step further; most of an LLM's training is next-token prediction on random internet content. A newly-trained LLM will just continue whatever text appears in its context window, like an extremely capable autocomplete. The illusion of an entity that takes turns in conversation and presents a consistent personality is tacked on at the last minute through RLHF. This was the transition from GPT to ChatGPT.

Any positive evidence of LLM consciousness should probably mostly be taken from the model before post-training, where it displays remarkable capabilities but shows no sign of a consistent personality, and likely no signs of self-awareness or self-understanding.