And AI is likely a binary outcome. It’s either very good for us or not, and it won’t take a million years to figure out which. We were in caves not long ago.
Very little to no evidence that continuously increasing intelligence will maintain the status quo indefinitely, on a scale of at least thousands of years. The odds of us not either populating the galaxy or utterly owning ourselves are vanishingly small. What is the middle ground that you think is likely, and what things have to happen for that to be true? Why is it more likely than one of the extremes?
It's so interesting that our intuitions are vastly different on this. You and I both just can't believe that our default case isn't obvious.
I personally think "recency bias" is to blame for the "we'll muddle through" case. Life is great and the weather is fine, and there are no asteroids or globally impacting volcanoes, and we haven't had a civilisational collapse in a good while, and nukes didn't kill us, so the current state is pretty locked in indefinitely, despite massive technological change that we have zero chance of predicting the outcomes of.
That's an unstable equilibrium at best. We take over the galaxy or die trying.
Well we are pretty capable of predicting, right now, that our civilization is very likely to collapse in the next few decades because we can't seem to address the climate/biodiversity problems (which are consequences of the abundance of fossil fuels that will end soon).
Loss of keystone species[1]. Say, if bees went extinct, our agriculture would come crashing. We still rely on other organisms for a lot of important, sometimes poorly understood until too late[2] functions that sustain our civilization.
If bees went extinct, it would affect none of the essential crops we grow. Grains are wind pollinated, not insect pollinated.
More generally, I'm sure extinctions would wreak havoc on wild ecosystems. But we grow food by explicitly destroying wild ecosystems, replacing them with monocultures. Crops seem to grow just fine anyway.