[startupsfail]

When doing the processing GPT remembers these in the “working memory” (very similar to your working memory that is just an actuation of neurons, not an adjustment of the strengths of synaptic connections).

And then, there is a chance that the inputs and outputs of GPT be saved and then used for fine-tuning. In a way that is similar to long-term memory consolidation in humans.

But overall, yes, I agree, GPT-4 in an empty space, without fine-tuning is very limited.

[dilyevsky]

It doesn’t remember anything unless you mean that intermediate values in calculation of forward pass is “remembering”. The prompt continuation feature is just a trick where they refeed previous questions/replies back to it with new questions at the end