I hear you. I think we are already seeing some middle ground with agentic systems using RAG, skills.md files, etc. It's a sort of disassociated card catalog memory. An engineer's notebook. Not the integrated, correlated, pre-processed set of relationships in the model. How to go backward from the notebook -> model cheaply without tanking performance is definitely one of those billion dollar questions.
a little glib, but there is in fact long term learning. It's just that you are not the one mentoring- the models go to intensive OpenAI/Anthropic/Google school for a quarter or half a year and come back (hopefully) improved. You just hope they're getting a good education. Certainly it's a very prestigious one.