[elmerfud]

It's an interesting question you pose here. For me I would think about it in the sense of how can this be used. If you're building something that can only be used for negative purposes that is something I would not want to be a part of and I would want others to know about it regardless of any NDA that I may have signed or other agreements. This can be a tough spot to be in but I believe it is the morally correct thing to do just as Edward Snowden did the morally correct thing but the legally incorrect thing.

And now if the project is kind of in nebulous as to what it's ultimate uses could be for that's a different situation entirely. Hammers can build houses or they can hurt people. With a generic tool you must trust the people to be able to use it responsibly and understand that some people will not use it responsibly. Additionally your refusing to help is unlikely to stop the project or promote people to only use it wisely.

I am not personally concerned about AI like some people are because it is merely a tool and people have regularly shown that they can use tools very well or they can use tools very poorly and the introduction of AI does not change that fact about human beings. If this is really a groundbreaking thing in a way that nobody can comprehend except those directly working on it the one thing to consider would be the impact of it being isolated to one group or one company or one government. When developing a radical game-changing technology that can be leveraged and weaponized in an extreme way you must seriously evaluate how you can trust the handlers of that technology. Do those handlers have the moral authority and have they demonstrated that to use it wisely and prudently and can you trust that in the future they will as well.

If it is truly revolutionary in an unexpected way you could refer to the lessons of history around the atomic bomb. Certain scientists who worked on that project feared having it only be available to one set of countries that were ideologically different than another set of countries. They feared it so much that they felt the need to provide balance in the playing field. No one knows what would have happened if that technology remained private for 10 or 20 years longer than it did would the United States have turned into a war mongering Nation or would it have used restraint and simply waged an idiological cold war? We will never know the answer to that but if it is truly revolutionary on that scale that it can cause a global imbalance in power between major Nations you must consider those possibilities.

Since you asked what would I do if I was faced with the idea that I was working on a piece of technology that could radically change the balance of power for my company and more generally for my country which is the United States, I'm not sure. If it could be used to kill people on a massive world scale with little to no reprocussions I would trust that the United States would not do that. The United States loves to be seen as the good guy even though we're not perfect we do love to promote that image. Now if it could be used in an information warfare campaign to spy on people to decrypt information to manipulate and control markets in a way that is not readily perceptible I would not trust them at all. The United States has shown repeatedly it is more than willing to violate the constitutional rights have its citizens that it's eager to declare people military combatants when it suits them and then toss them into a civil trial when it does no longer suits them. You can do these things because they can do them all behind closed doors where they're almost never seen or heard about.