[reliablereason]

Unless you enforce your network to have simple understandable mechanics during training you won’t be able to properly decompose the network to a mechanistic understandable explanation. Due to the fact that it won’t work like that under the hood. Parts of the network will be simple enough to be understandable but allot of it won’t.

The human mind can not deal with and understand things that have to much dimensionality.

My 5 cents.

[DanielBMarkham]

I agree, but keep in mind that the goal of AI (whatever that is) all along is not explanatory power, it's verisimilitude. This is due to necessity, since we don't understand it already, the only way to judge it is to see if it kinda sorta makes sense.

Looked at this way, this project is a thing of beauty. Just like the human mind, it's not going to explain how the LLM came up with an answer, it's going to create plausible arguments why it did so. How would we know? Do you actually think it'll make a difference whether the AI is correct about any random oddball question, especially when it can make a pretty coherent case about why the answer was correct all along?

It will not.

We are performing a very strange experiment with our fellow humans. Should be fascinating to watch it play out.