[pembrook]

You bring up the example of an extinction-level asteroid hurling toward earth. Gee, I wonder if this superintelligence you’re deathly afraid of could help with that?

This extreme risk aversion and focus on negative outcomes is just the result of certain personality types, no amount of rationalizing will change your mind as you fundamentally fear the unknown.

How do you get out of bed everyday knowing there’s a chance you could get hit by a bus?

If your tribe invented fire you’d be the one arguing how we can’t use it for fear it might engulf the world. Yes, humans do risk starting wildfires, but it’s near impossible to argue the discovery of fire wasn’t a net good.

[yard2010]

Since the internet inception there were a few wrong turns taken by the wrong people (and lizards, ofc) behind the wheel, leading to the sub-optimal, enshitified tm experience we have today. I think GP just don't want to live through that again.

[immibis]

You mean right turns. The situation that we have today is the one that gets most rewarded. A right move is defined as one that gets rewarded.

[voidspark]

I think of the invention of ASI as introducing a new artificial life form.

The new life form will be to humans, as humans are to chimps, or rats, or ants.

At this point we have lost control of the situation (the planet). We are no longer at the top of the food chain. Fingers crossed it all goes well.

It's an existential gamble. Is the gamble worth taking? No one knows.