[js8]

> Wondering if this article is written by GTP-3.

FYI, it was written by a woman. I looked up her book, Kill It with Fire, and as a mainframer I have to say it seems pretty interesting.

I think what she alludes to in this essay, though, is more like that AI cannot solve socioeconomic problems of humans. And even humans seem to struggle with it.

Whenever I read stories where the metrics became the targets, and the like, I am reminded of Varoufakis' book Economic Indeterminacy. He doesn't give any answers there, but there is this "strange loop" in rationalism that nobody really understands.

I also think that AI might be a wrong target, because you need to understand the problem before you can solve it, and once humans understand the problem, they don't need AI anymore, they just code the solution as an algorithm. On the other hand, if humans don't fully understand the problem, it's extremely difficult (except artificial circumstances like games) to explain to AI what the problem is, so it would arrive at a "reasonable" solution (and avoided, at the very least, killing all humans).

[soco]

If we can't understand the problem, will we be able to understand the solution presented by that AI? Or we'd just apply it, trusting blindly the unfathomable reasons the AI used? Do we have an AI where the decision tree can be grasped by humans?

[lokischild]

It seems inevitable to me the moment AI capacity seriously surpasses human (as a whole) capacity in any specific topic, it becomes an oracle. I hear of efforts to translate machine decision making to human understandable terms, but if it is a question of raw intelligence, it will quickly become impossible to understand.

[maest]

It's interesting to see how super advanced ML-based engines have changed the field of chess in recent years - top engines are way better than top humans, so, in a sense, act as oracles.

Top players routinely use them in preparation, to study new lines, get hints about what moves make sense in a position and also to generate new ideas or tweak the principles they apply in the game. The engines don't explain their reasoning, but provide something closer to the "correct" move in any given position. It's up to the humans to do the legwork and understand _why_ the recommended move is strong.

Clearly, chess is not real life, but the impact of these oracle engines has been broadly positive (with the exception of using engines to cheat in online play).

[lokischild]

This is a good example, because Chess rules do not surpass human intellectual capacity. With enough time, you can understand the reasoning of an AI.

But when it comes to raw intellectual superiority, say, explain the logic behind new mathematical or physical concepts and discoveries, very high level predictions, we come to our human limit and cannot surpass it.

The AI can give us implants to improve our intelligence, but only to a certain limited degree, our biological brain will slow us down significantly.

That is the barrier we cannot possibly cross in a timespan of human life.

[js8]

This is a topic of Lem's novel/essay https://en.wikipedia.org/wiki/Golem_XIV.

And to be honest, I wondered about that with GPT-3. Maybe it could give a more profound answer to a given prompt, but it chose not to, since it found the prompt to be too silly, and it responded in kind. Just like adults are able to entertain an imaginary universe of children.

So even to explain that we want a "serious" answer might be difficult.

[amelius]

> I think what she alludes to in this essay, though, is more like that AI cannot solve socioeconomic problems of humans.

Yes, the question is: do we want AI to solve first world problems, or real problems?