[mft_]

Thanks for sharing your experience. It's really interesting that you describe a loss of some 'middle' parts but not others. The 'classic' medical/psychological model of memory has three parts (sensory/short-term/long-term), but it's also worth noting that that model was first devised in 1968!

> long term memory of experiences. You can fake this with memory.md

Not sure about this; to my mind, memory.md is analogous to humans making lists of things to not forget to do, or notes from a lecture to learn (i.e. cram into long-term memory) later on. LLMs use it as a short cut to bring important facts back into their context window; but it's not the same as them already 'knowing' the information via the original training process.

---

My consistent (hot?) take is that a (the?) major piece holding LLMs back (maybe even from AGI?) is continual learning. Humans have systems for continually updating their long-term memory from their lived experience - new facts, processes, skills, successes, mistakes, etc. (Sleep and dreaming are centrally involved in this process.) The current architecture of LLMs makes this practically impossible, as it would presumably require the level of power currently necessary for training to be continually applied for continual learning, and demonstrates the huge efficiency advantage of the biological brain.