[ben_w]

A large part of that goes via arguing about consciousness.

Starts of with a quite defensible:

  First: super-intelligent AI is unlikely because, if you pursue Vernor's program, you get there incrementally by way of human-equivalent AI, and human-equivalent AI is unlikely.

But the moment he hits that (parenthetical) paragraph about consciousness, he blends human capabilities with human-like consciousness.

If human-like capabilities required human-like consciousness, early models (like GPT-4) passing all those exams would suggest those models being "as conscious as" a university student. (My inner Douglas Adams wants to invert this, but the best I can do is "The Xarloxians did not consider university students to be conscious beings; their reasoning being…", which you may recognise as a riff on Babel fish and the non-existence of God).