[halosghost]

I found this logical leap to be pretty straight-forward, but it is a leap (rather than a clearly-telegraphed step).

My understanding was that the author sees the following:

  - Lots of AI advocates don't just think an AI superintelligence is possible, but desirable
  - They reason that an AI superintelligence will guide humanity to become better, and more intelligent
  - There is no known, proven, objective measure of “intelligence,” all schools so far that espouse such a thing as being possible and coherent stem from eugenics, phrenology, and similar projects (VC, AI founders, and CEOs have significant overlap with these groups)
  - AIs reinforce the biases present in their training sets and tuning, meaning they will gladly spout rhetoric from those same sociopolitical projects
  - People believe AI output readily and blindly

Regardless of whether or not a Superintelligence emerges (I'm betting against), the outcome of this trajectory will be that the eugenics/phrenology ideological descendants will continue to guide any new AI to espouse their beliefs (whether intentionally or due to implicit bias), and that most people will just accept the resulting outputs as True™. AI then will reinforce eugenicist views (as it already is). ∎

All the best,