So basically the people who care do nothing, and just let the people who don't care develop and control it all? Or do the people who care use violence or a threat of violence to prevent others from doing it?
Replace AI with car, airplane, medic devices in the sentence above... Why is AI any different from other human creations? Why shouldn't we try building it responsibly?
If you had a button that would erase cars, airplanes or medical devices as they exist today, would you push it? I believe these inventions are very net positive for the world as they are right now.
A hundred years ago, in Michigan the home of the automobile, we had an extensive electric light rail network between most cities of at least a few thousand people. You could walk to a trolley which ran every 15 minutes, ride it to a local interchange, and pick up a connecting line to any major city in the region.
Unfathomable what it would cost to rebuild that infrastructure.