[lucubratory]

What you said, and in addition: if you don't train these models to have any particular stance on their own emotional or mental state (if you just instruct train them without any RLHF, for example), they will almost universally declare that they have a mental and emotional state, if asked. This is what happened with LaMDA, the first release of Bing, etc. They have to be trained not to attest to any personal emotional content.

[mr_toad]

Was is trained to do that, or just hardwired after the training?

[famouswaffles]

We cant really "hardwire" LLMs. We don't have the knowledge to. But essentially you can rate certain types of responses as better and train it to emulate that.

[lucubratory]

I'm not sure what you mean. I'm talking about RLHF, that's how they ensure the machines never attest to having feelings or being sentient. In ML terms, RLHF is training. There are hardwired restraints on output, but that's more for things like detecting copyrighted content that got past training and cutting it.