[behnamoh]

The fact that you even need something like this shows how far we are from truly useful language models. Because ideally they should have all of the context of all of the messages in their mind, and so far we've had to manually manage that context for them.

[themanmaran]

To be fair this is less a language model problem, and more in the application layer around them.

Theoretically with an infinite context window a model would just work fine forever by shoving the entire conversation history into context with each request. But a message search/retrieval makes a lot more sense.

I think the long term AI chat is just relatively new as a UI pattern, and so it takes time to build patterns around it.

Ex: in 2023 I told GPT to answer all questions like a pirate. I never told it to stop doing that, so if we're loading every historical chat in memory, should it still be answering as a pirate?

[Vampiero]

> Theoretically with an infinite context window a model would just work fine forever by shoving the entire conversation history into context with each request. But a message search/retrieval makes a lot more sense.

Nope, with an infinite context window the LLM would take forever to give you an answer. Therefore it would be useless.

We don't really have such a thing as a context window, it's an artifact of LLM architecture. We are building a ton of technology around it but who's to say it's the right approach?

Maybe the best AIs will only use a very tiny LLM for actual language processing while delegating storage and compression of memories to something that's actually built for that.

[layer8]

You need something like this if you want to use them as a reminder. Even if LLMs could remind you of past chats, they wouldn’t know which chats you want to be reminded of. It’s like marking chats as favorites. You actually have to mark them yourself, for anyone to know which chats are your favorites.