[nerdponx]

Right, that this can happen should be obvious from the transformer architecture.

The fact that these things work at all is amazing, and the fact that they can be RLHF'ed and prompt-engineered to current state of the art is even more amazing. But we will probably need more sophisticated systems to be able to build agents that resemble thinking creatures.

In particular, humans seem to have a much wider variety of "memory bank" than the current generation of LLM, which only has "learned parameters" and "context window".