[ben_w]

> But training just allows it to replicate what it's seen.

Two steps deeper; even a mere Markov chain replicates the patterns rather than being limited to pure quotation of the source material, attention mechanisms do something more, something which at least superficially seems like reason.

Not, I'm told, actually Turing compete, but still much more than mere replication.

> It's the same when I have a conversation with it, then tell it to ignore something I said and it keeps referring to it. That part of the conversation seems to affect its probabilities somehow, throwing it off course.

Yeah, but I see that a lot in real humans, too. Have noticed others doing that since I was a kid myself.

Not that this makes the LLMs any better or less annoying when it happens :P