[0xBA5ED]

Well, in order to explain why you did a thing, you need to have some understanding of how the other party thinks in order to explain it in a way they will understand. And in order to explain it in that way, you would need to reflect upon the meaning of your own decisions in the context of how the other party models things.

[MiracleUser]

Yes I think we would need a design goal to provide the AI with "some understanding" of how we think. It should be able to do its work in any model that works for it, which would then be translated back to our language.

Thankfully, language is flexible so it can express many models we do not understand in ways that can help us understand them. It would be interesting to see some sort of study on where the boundaries are on what human language is capable of modelling or not