[skohan]

The problem is who is "we"?

If the capability exists, someone will pursue it. Maybe the G20 will pass a resolution to put safety measures in place, but whoever decides to flaunt those safety measures stands to gain an asymmetrical power advantage

That sets up a competitive dynamic where players are incentivized to try to get there first, no matter the risks, because if they don't someone else will

Similar dynamics played out with the nuclear arms race

[vouaobrasil]

"We" refers to human civilization, and is not just one CEO but the sum of all society that acts as an organism through the emergent behaviour of our social existence.

If "we" are doing something wrong, then "we" had better fix it. And that could mean governments or even counter-actions by individual people, or even actions by hacker groups....

[skohan]

Yes I understand that, but the problem is there's no structure to enforce decisions at the level of human civilization. So even if something is agreed and enforced by 99% of jurisdictions, that doesn't mean it won't happen

[vouaobrasil]

And it also doesn't mean individuals can't revolt against it either. It could happen, and 1% might try and make it happen, but we can ALSO try and stop that 1%, with a mob if necessary.

[thereisnospork]

Yes it does: because all any isolated revolt can do is hurt your side[0].

[0]whomever you are wanting to revolt on behalf of, and you can't revolt on behalf of everyone.