What's concerning about this is we are evaluating AI on a basis that humans are not subject to. LLMs in their current form are built on the knowledge of the internet, while humans have both the internet and realtime feedback from their own lives in the physical world. If a human brain could be trained the same way as an LLM, might it also connect seemingly unconnected ideas in a way that would appear as non-thought? Maybe, maybe not. LLMs seem to be biased heavily towards making best effort guesses on things it doesn't know about, whilst humans are far more modest in doing so. I just don't know if we're really at a point where we can conclusively decide that something isn't thinking just because it doesn't appear to be thinking by the standards we place upon ourselves.