[chongli]

Software is not the world. It’s a tiny bit of what humans do.

The lack of online learning is a critical fault. Much of what humans learn (such as anything based on mathematics) has a dependency tree of stuff to learn. But even mundane stuff involves a lot of dependent learning. For example, ask an LLM to write a cookbook and it can synthesize from recipes that are already out there but good luck having it invent new cooking techniques that require experimentation or invention (new heat source, new cooking utensils, etc).

[sally_glance]

I guess we'll just have to wait and see how things turn out. Currently it seems we have examples of where it seems like the technology allows some amount of innovation (AlphaGo, software, math proofs) and examples where they seem surprisingly stupid (recipes?).

Btw, it looks like there is a growing body of research evaluating exactly this. I found this nice overview with even some benchmarks specifically for scientific innovation: https://github.com/HKUST-KnowComp/Awesome-LLM-Scientific-Dis...