[MiracleUser]

I'm not sure what you mean. Why would a foreign model inherently have the attribute of being inexplicable?

Isn't it more likely that we would look at a foreign model and think, "this makes sense, but i wouldnt have ever thought to do it this way".

Our senses might be fooled, but I think there would almost always exist a set of words to explain what the AI is doing assuming the AI is capable enough to find them

[0xBA5ED]

Well, in order to explain why you did a thing, you need to have some understanding of how the other party thinks in order to explain it in a way they will understand. And in order to explain it in that way, you would need to reflect upon the meaning of your own decisions in the context of how the other party models things.

[MiracleUser]

Yes I think we would need a design goal to provide the AI with "some understanding" of how we think. It should be able to do its work in any model that works for it, which would then be translated back to our language.

Thankfully, language is flexible so it can express many models we do not understand in ways that can help us understand them. It would be interesting to see some sort of study on where the boundaries are on what human language is capable of modelling or not