An exec can ask an expert to explain themselves, and if they're a good CEO they can weigh the pros and cons of the arguments from their staff and make a decision. Someone that's much smarter or more skilled in a particular discipline can usually explain what's going on to someone that's not as smart as long as the gap isn't too big. However, I've found that with Fable, and some of the latest frontier models their critique of my ideas is brilliant but on the edge of what I can even comprehend. I am above average intelligence, so I am sure a lot of people with less than average intelligence already can't understand these understandings it's formulated, and instead will just default to, "you know what you're doing, whatever." There will be the case in the future till the brilliance is so complicated that all of us will be doing this, and then we'll just have to trust the AI.
For example, I could imagine a future AI telling me that it has modeled my behavior and built a very large differential equation that seems to perfectly fit my ideal pattern to maximally achieve my life goals and it looks like something Ramanujan came up with, and I'd tell it "that looks great, let's optimize my life based on that" while having no ability to even approach understanding something like that.
Can it dumb down Ramanujen equations so someone with average intelligence could understand them if even world class mathematicians struggled to grasp them? This is the best analogy for super intelligence. Alpha Go can't really explain why it makes the go moves it does in a way a human can understand. Sure it could show them all the weights in the model, but that's not anything a human can understand.
For example, I could imagine a future AI telling me that it has modeled my behavior and built a very large differential equation that seems to perfectly fit my ideal pattern to maximally achieve my life goals and it looks like something Ramanujan came up with, and I'd tell it "that looks great, let's optimize my life based on that" while having no ability to even approach understanding something like that.