[orbital-decay]

A safety plan. It doesn't have to be "a few very smart people detached from reality convincing themselves they're messiahs that must keep the tech from the Bad Guys(tm), unwashed masses, and a runaway, because they think their interpretation of their own sci-fi lore is the only possible course of events"

>I hope you'll keep this in mind when those milestones are reached.

The problem is that people in charge of AI keep making self-fulfilling prophecies. Just like with any research, if you want to find something sensible in the cloud patterns, you will.

[SpicyLemonZest]

What is an example you have in mind of a self-fulfilling prophecy? I genuinely don't know what you could be referring to. It seems to me that they keep making surprising prophecies, and the popular reaction to them seamlessly transitions from "that's crazy, no way it will happen" to "that's silly, it's just a cloud pattern". Did you find it obvious or self-fulfilling in 2025 that LLMs would soon be able to resolve open questions in mathematical research?

[orbital-decay]

Self-fulfilling prophecies are social effects, not real predictions. Anthropic's "We predict our model misbehavior potentially being able do destroy the world in 1% of the cases" -> "We want to find the evidence of our model misbehaving, and we want it bad" -> "See, our model is hacking our rewards and has functional emotions, this means it's misbehaving with the intention of destroying the... HUMANITY!1" -> repeat x100, manipulate the media into amplifying it x10000 for clicks -> people are begging to safeguard them from the evil AI. Which is already likely to happen, the average layman's Overton window already includes the fantasy of rogue AI.

None of that was real or remotely dangerous in the first place, of course. It wouldn't have resulted in controls, had they not been scaremongering. This will end in extreme fascism or people getting enslaved "for their own safety", and it won't even require malicious intent, only incentives, detachment from reality, and confirmation bias. Although it doesn't exclude malice either.