[ChuckMcM]

I love that book, that said, the point is more subtle than that. Current LLM attention models are limited in their feedback. Adding a form of 'shame' feedback (result is technically correct but morally bad or some such) would help here but I doubt the folks building theses things would choose to do so.

[jerf]

From a certain and quite valid point of view, they have no mechanism for feedback at all. Every time you start a conversation you're starting in the same state, modulo the random numbers. At most you have this very, very vague loop in that the conversations for LLM 1.0 will be fed in to the training set for LLM 2.0.

Even "shame" would only apply to the current session and disappear in the next one, or eventually be compacted away.

(Although honorable mention to Gemini's meltdown: https://x.com/AISafetyMemes/status/1953397827662414022 )

[suzzer99]

According to ChatGPT, researchers are working on models that remember personal directives across sessions. IE - an actual personal assistant that gets to know you and your proclivities. So it's definitely on their radar. No idea how far along they are.

[jerf]

Unless that's something more than the already-common practice called "memories" that are text files held off to the side, that doesn't change what I meant. You can do all sorts of interesting things within the context window, but there's no feedback beyond that.

Even if an frontier-LLM-sized neural net could do something that would somehow change its net on a pervasive level in response to things that happen to it, nobody could possibly serve that in a cost-effective manner.