[totally_human]

Spying on your own citizens enables certain sorts of anti-democratic abuses (and has been used that way in the past), so I can understand the specific opposition to it. Put somewhat melodramatically, they're okay with spying but don't want to create self-coup tools.

I agree that the killbots red line is somewhat odd, but I guess you have to draw the line somewhere, and I prefer them having that principle to having no principle at all. (Also, it's possible that the AI insiders understand something I don't about why a human in the loop is important.)

[slopinthebag]

I would argue that isn't really a moral argument though, it's rather utilitarian. If someone at OpenAI disagreed with that risk assessment, that's a difference of opinion not a reason to quit and write letters talking about ethics.

Also it's a rather American-centric view. If a Canadian is working at OpenAI, should they care? Or would they care more about possible anti-democratic interference by the American government on Canada?

[totally_human]

Utilitarianism is a moral system. If you disagree with the risk assessment and are utilitarian, you believe that OpenAI got it wrong and is thus doing bad stuff. If a company making bridges, say, or nuclear power plants, was doing risk assessments that appeared to ignore substantial risks in order to get a lucrative contract, I would fully expect engineers to quit and start writing letters talking about ethics.

Agreed on the America-centric view, to an extent. I will note that almost all countries have spied on each other since time immemorial, but serious efforts to spy on their own citizens tend to coincide with uniquely repressive and unpleasant regimes. I think having a norm against spying on your own citizens is good, even if it isn't a perfectly elegant principle. Also, countries can do more damage spying on their own citizens vs other citizens -- as a Canadian, I don't want the American government spying on me, but I'd probably be more worried if the Canadian government was spying on me.