No one can even say what “that” is that we shouldn’t build. It’s a wholly arbitrary invention, the product of a collective word game that is more apologetics than serious futurology.
That's partially the point. We may not know it until it's too late.
But it doesn't matter. If you gave everyone a button and the warning that pushing it would end the human race, we'd all be dead pretty much immediately.
I think it's a given that on a long enough timescale, someone will create AGI and we'll no longer be the dominate race on earth. How that ends, in death, or slavery, I don't know.
I do hope that every AI researcher does their work with a small bit of fear of becoming the pet of an AGI.
I actually don't think we should build it either. The best case scenario is it provides a way for powerful people to accelerate the already quite successful project of making feudalism 2.
But that doesn't mean I'm stressed about what you're stressed about. We killed god once before we can do it again if we need to. The real danger is in us using AI to do more of what it already does now, not that it will decide to do some new thing.