One could train an LLM on pre-Einstein, or pre-Newton, material, and see if it can "invent" the laws of gravity or relativity. If enough care is done in avoiding contaminating the training corpus, that should settle the question.
That would be a fascinating experiment. If an LLM trained only on pre-Newton or pre-Einstein material independently arrived at something resembling those theories, it would be strong evidence that powerful new representations can emerge from existing information.
That’s an interesting point. Brains and LLMs are certainly very different in their implementation. My argument, though, is less about the implementation itself and more about the fact that both systems are capable of generating and manipulating representations.