[jltsiren]

It's a well-established principle in computer science that the input/output behavior of a system may not capture all of its important properties. Take zero-knowledge proofs for example. Their entire point is that they are indistinguishable from randomly generated garbage from a specific distribution. The proofs only gain value if you make causal assumptions about the system that generated them.

I don't think systems like GPT-4 can ever be truly intelligent, because they simply output randomly generated garbage from a specific distribution. Their output may eventually be indistinguishable from that of a truly intelligent system, but the causal mechanism behind them is not intelligent.

On the other hand, most people lose their ability to think when they are under sufficient pressure (such as fighting for their lives). It's plausible that people are fundamentally no different from systems like GPT-4 in such situations. Then a language model could be a key part of an AGI, but true intelligence would also need higher-level causal mechanisms.