No one can even say what “that” is that we shouldn’t build. It’s a wholly arbitrary invention, the product of a collective word game that is more apologetics than serious futurology.
That's partially the point. We may not know it until it's too late.
But it doesn't matter. If you gave everyone a button and the warning that pushing it would end the human race, we'd all be dead pretty much immediately.
I think it's a given that on a long enough timescale, someone will create AGI and we'll no longer be the dominate race on earth. How that ends, in death, or slavery, I don't know.
I do hope that every AI researcher does their work with a small bit of fear of becoming the pet of an AGI.
But it doesn't matter. If you gave everyone a button and the warning that pushing it would end the human race, we'd all be dead pretty much immediately.
I think it's a given that on a long enough timescale, someone will create AGI and we'll no longer be the dominate race on earth. How that ends, in death, or slavery, I don't know.
I do hope that every AI researcher does their work with a small bit of fear of becoming the pet of an AGI.