[ArcaneMoose]

Interesting point - thanks for sharing! I think one big missing piece we have with AIs today is the ability for them to learn on the fly and reconfigure the weights. We are constantly bombarded with input and our neurons adjust accordingly. Current LLMs just use a snapshot. I would be really curious to see how online-first AI models could work, focusing on a constant input stream and iterating on weights. Also I wonder how much knowledge is baked into our DNA through evolution. I have a hunch that this is somewhat analogous to model architectures.

Btw - although I see evidence of LLMs creating "new ideas" through combinations of ideas, I am a bit mystified by their apparent reasoning issues. I wonder how that is different in nature from the memory-based approach. ARC-AGI benchmark has had me thinking about this for sure.