[soco]

If we can't understand the problem, will we be able to understand the solution presented by that AI? Or we'd just apply it, trusting blindly the unfathomable reasons the AI used? Do we have an AI where the decision tree can be grasped by humans?

[lokischild]

It seems inevitable to me the moment AI capacity seriously surpasses human (as a whole) capacity in any specific topic, it becomes an oracle. I hear of efforts to translate machine decision making to human understandable terms, but if it is a question of raw intelligence, it will quickly become impossible to understand.

[maest]

It's interesting to see how super advanced ML-based engines have changed the field of chess in recent years - top engines are way better than top humans, so, in a sense, act as oracles.

Top players routinely use them in preparation, to study new lines, get hints about what moves make sense in a position and also to generate new ideas or tweak the principles they apply in the game. The engines don't explain their reasoning, but provide something closer to the "correct" move in any given position. It's up to the humans to do the legwork and understand _why_ the recommended move is strong.

Clearly, chess is not real life, but the impact of these oracle engines has been broadly positive (with the exception of using engines to cheat in online play).

[lokischild]

This is a good example, because Chess rules do not surpass human intellectual capacity. With enough time, you can understand the reasoning of an AI.

But when it comes to raw intellectual superiority, say, explain the logic behind new mathematical or physical concepts and discoveries, very high level predictions, we come to our human limit and cannot surpass it.

The AI can give us implants to improve our intelligence, but only to a certain limited degree, our biological brain will slow us down significantly.

That is the barrier we cannot possibly cross in a timespan of human life.

[js8]

This is a topic of Lem's novel/essay https://en.wikipedia.org/wiki/Golem_XIV.

And to be honest, I wondered about that with GPT-3. Maybe it could give a more profound answer to a given prompt, but it chose not to, since it found the prompt to be too silly, and it responded in kind. Just like adults are able to entertain an imaginary universe of children.

So even to explain that we want a "serious" answer might be difficult.