[moffkalast]

I mean I can definitely remember lots of cases for myself, in school especially, when I made the same mistake again repeatedly despite being corrected every time. I'm sure today's language models pale in comparison to your flawless genius, but you seriously underestimate the average person's idiocy.

Agreed that the lack of some mid tier memory is definitely a huge problem, and the current solutions that try to address that are very lacking. I highly doubt we won't find one in the coming years though.

[somenameforme]

It's not just this. LLMs can do nothing but predict the next token based on their training and current context window. You can try to do things like add 'fact databases' or whatever to stop them from saying so many absurd things, but the fact remains that the comparisons to human intelligence/learning remain completely inappropriate.

I think the most interesting thought experiment is to imagine an LLM trained on state of the art knowledge and technology at the dawn of humanity. Language didn't yet exist, slash 'em with the sharp part was cutting edge tech, and there was no entirely clear path forward. Yet we somehow went from that to putting a man on the Moon in what was basically a blink of the eye.

Yet the LLM? It's going to be stuck there basically unable to do anything, forever, until somebody gives it some new tokens to let it mix and match. Even if you tokenize the world to give it some sort of senses, it's going to be the exact same. Because no matter how much it tries to mix and match those tokens it's not going to be able to e.g. discover gravity.

It's the same reason why there are almost undoubtedly endless revolutionary and existence-altering discoveries ahead of us. Yet LLMs trained on essentially the entire written corpus of human knowledge? All they can do is provide basic mixing and matching of everything we already know, leaving it essentially frozen in time. Like we are as well currently, but we will break out. While the LLM will only move forward once we tell it what the next set of tokens to mix and match are.

[Jensson]

It lacks more than memory, it makes the mistake again later even when the previous prompt is in its current token limit.

[moffkalast]

Sure, it happens. How often it happens really depends on so many factors though.

For example, I have this setup where a model has some actions defined in its system prompt that it can output when appropriate to trigger actions, and the interesting bit is that initially I was using openhermes-mistral which is famous for its extreme attention to the system prompt, and it almost never made any mistakes when calling the definitions. Later I swapped it with llama-3 which is way smarter, but isn't tuned to be nearly as attentive and far more often likes to make up alternatives and don't get fuzzy matched properly. Someone anthropomorphizing it might say it lacks discipline.