The problem is that we don't even know what might be there to understand. Since it is pretty much unknown how humans think (from a brain processing perspective), we also don't really know if the things we see in LLMs are just dumber versions of human thought processes or if they are qualitatively different.
LLMs literally can’t compute any complex algorithm, they can’t simulate different steps in chess to get which version is best, nor execute some non-trivial algorithm.
They are nowhere near close to human intelligence.