[nprateem]

But training just allows it to replicate what it's seen. It can't reason so I'm not surprised it goes down a rabbit hole.

It's the same when I have a conversation with it, then tell it to ignore something I said and it keeps referring to it. That part of the conversation seems to affect its probabilities somehow, throwing it off course.

[nerdponx]

Right, that this can happen should be obvious from the transformer architecture.

The fact that these things work at all is amazing, and the fact that they can be RLHF'ed and prompt-engineered to current state of the art is even more amazing. But we will probably need more sophisticated systems to be able to build agents that resemble thinking creatures.

In particular, humans seem to have a much wider variety of "memory bank" than the current generation of LLM, which only has "learned parameters" and "context window".

[swax]

Humans are also trained on what they’ve ‘seen’. What else is there? Idk if humans actually come up with ‘new’ ideas or just hallucinate on what they’ve experienced in combination with observation and experimental evidence. Humans also don’t do well ‘ignoring what’s been said’ either. Why is a human ‘predicting’ called reasoning, but an AI doing it is not?

[nprateem]

Because a human can understand from first principles, while current AIs are lazy and don't unless pressed. See for example, suggesting creating bleach smoothies, etc.

[ben_w]

> But training just allows it to replicate what it's seen.

Two steps deeper; even a mere Markov chain replicates the patterns rather than being limited to pure quotation of the source material, attention mechanisms do something more, something which at least superficially seems like reason.

Not, I'm told, actually Turing compete, but still much more than mere replication.

> It's the same when I have a conversation with it, then tell it to ignore something I said and it keeps referring to it. That part of the conversation seems to affect its probabilities somehow, throwing it off course.

Yeah, but I see that a lot in real humans, too. Have noticed others doing that since I was a kid myself.

Not that this makes the LLMs any better or less annoying when it happens :P