| My mental model is this: There is a vast ocean of human knowledge, far beyond the capacity of any human brain, even within specialised fields. Books helped "plug the gaps" in our knowledge, increasing the scope that a single human mind can encompass. Web search engines did the same thing, but more and faster. LLMs are like search engines on steroids, essentially a research librarian that operates at 1,000x human speed and can "in context" locate relevant information, adapting it to fit the hole it needs to go into as well. It feels less like discovering new theorems, but instead having direct access to all theorems, which is hugely valuable in itself. I.e.: the recent counterexamples to open conjectures has largely been about the AIs "trawling through all the things" and scraping together every bit of human-generated knowledge ever produced that is relevant to the conjecture. Conversely, in the past, we had to "make do" with sub-standard solutions where the problem had been solved, but finding every relevant solution in the ocean of knowledge was prohibitively time consuming. In some sense, LLMs will "raise the floor" in what is considered the minimum level of quality of a solution, where even throwaway / toy designs will now start applying every bit of accumulated wisdom instead of just some of it. We have mechanised attention. |
The rare benchmarks that measure "knowing what the model knows it doesn't know" show us there are only a couple models like Opus that are good in that field.
I think this is something that receives not enough attention from researchers.