[lyfeninja]

I noticed chatGPT using 1.4GB in a single chat the other day and affecting performance. Don't get me wrong it was a long one, but it's all text, so I still don't think I understand why it uses so much memory. A novel should only take up a few MBs of memory. I assume it's a ton of backend context, but feel like someone should look into that. No wonder all the robots are greedy for memory.

[muzani]

ChatGPT makes sense though. They shouldn't be dumping the whole chat into memory the entire time. They're compressing it in some way. If they do it on device, it saves the cost of doing it on the cloud. ChatGPT's memory feature is a good leap ahead of the competition and it could be due to things like this, which may query memories of conversations made long ago and not just the recent ones.

Tools like Cursor do something very similar. They claim to be using 2 million tokens or something, but those are cheap tokens which make the code space more searchable.