[gwern]

The Turing machine argument (which Scott Aaronson has also made) for superintelligence being comprehensible is pointless because you wouldn't have time to run the TM to completion (look up how many FLOPS it takes to run even merely a GPT-3 model for a conversation and estimate how long it'd take you to do that by hand), the Chinese Room argument applies (only the TM understands, you, a mere component, do not), and humans are too error-prone and backslide too much to achieve certain things. Why can't you teach the dumbest kid in your school calculus if he's equivalent to a Turing machine? Because he forgets too fast to learn anything on net! The further up in difficulty you get, the more time gets spent on review and relearning before any new material can be broached. I've tutored low-performing kids, and you can almost see them forgetting material as it slips away from them after a brief period of comprehension, fading like a dream upon awakening, leaving only frustration and dissatisfaction. Even more extreme example: my mother worked in special ed and told me about how each year higher you go with the kids, the more you have to review; especially in contrast to the mainstreamed kids. Unsurprisingly, this asymptotes at a low level. Perhaps the most extreme example are 'click' or threshold things: you can talk about dynamic programming to someone all you want, but if it hasn't clicked, it isn't there; many Ravens matrix style problems are like that; and Piagetian developmental stages are famous for that - if you are a kid who doesn't get that volume of water is conserved from a rectangular glass to a square glass, you don't get it.