[handoflixue]

Okay, so you've established that LLMs aren't programming. They are unlike existing machines. The closest analogy we have for them is the human brain, which also seems to include a lot of neural net architecture.

Now, physics says that everything can be explained mathematically, including the human brain. Obviously, on some level, an LLM can be explained. But despite hundreds of years of science, we still don't understand the human brain. Some systems are just really complex and difficult to understand.

Given all of that, I see no reason to assume that we'll be able to understand LLMs anytime soon. Especially given we keep growing more complex ones.