[themanmaran]

To be fair this is less a language model problem, and more in the application layer around them.

Theoretically with an infinite context window a model would just work fine forever by shoving the entire conversation history into context with each request. But a message search/retrieval makes a lot more sense.

I think the long term AI chat is just relatively new as a UI pattern, and so it takes time to build patterns around it.

Ex: in 2023 I told GPT to answer all questions like a pirate. I never told it to stop doing that, so if we're loading every historical chat in memory, should it still be answering as a pirate?

[Vampiero]

> Theoretically with an infinite context window a model would just work fine forever by shoving the entire conversation history into context with each request. But a message search/retrieval makes a lot more sense.

Nope, with an infinite context window the LLM would take forever to give you an answer. Therefore it would be useless.

We don't really have such a thing as a context window, it's an artifact of LLM architecture. We are building a ton of technology around it but who's to say it's the right approach?

Maybe the best AIs will only use a very tiny LLM for actual language processing while delegating storage and compression of memories to something that's actually built for that.