[Xeronate]

I understand the vision, but how does this work on a global scale. e.g. American employees refuse to build this, but China's don't.

Edit: I originally ended with "What would have happened if Germany had a nuclear bomb and America didn't?", but I think it distracted from the point I was trying to make so moving this to an edit. I'm not trying to ask "is the US the bad guy". I'm trying to ask how to balance personal anti war sentiments with the realities of the world (specifically in this case keeping up in an arms race).

[protocolture]

>American employees refuse to build this, but China's don't.

How about you articulate the threat from an AI powered China to people outside of AI powered China and discuss potential methods to counter that, instead of insisting capabilities be developed just in case.

>is the US the bad guy

Yes

>I'm trying to ask how to balance personal anti war sentiments with the realities of the world

Insist on open information, never surrender consent willingly and demand justification for everything. As always.

[boxedemp]

The threat seems straightforward to me; information warfare.

[wbobeirne]

Then the follow up question, how do you combat that? Not likely through developing similar technology.

[pjc50]

The US is a major exporter of that. Including Google itself via the YouTube recommendations algorithm.

[protocolture]

How has the threat model changed?

Keep your shit patched. You dont need LLM targeting of Drone based weapons to patch your servers.

[pjjpo]

There is no bad guy in geopolitics, just the other guy.

[protocolture]

"geopolitics" are an abstraction that tries to pave over guilt. But Abstractions are only useful if they provide some benefit. Geopolitics provide no benefit when assigning guilt so its easily ignored.

[remarkEon]

I think you've missed the point.

PRC isn't going to do any of the things you are asking for, and no one expects them to. The threat of an AI powered China is really obvious to me, but apparently the idea of "IP theft and industrial sabotage, but at scale with AI agents instead of human meat sacks" is hard to clearly articulate.

One method, beyond AI powered kill chains, to counter an AI powered China is of course strategic weapons.

[protocolture]

I think you missed the point.

I didn't ask China for anything.

[skybrian]

Not to worry, xAI would do it even if Google didn't.

Also, Anthropic didn't actually refuse to work on all military stuff. They have some conditions, which isn't the same thing.

[baq]

Anthropic has conditions, the Pentagon has billions of carrots and millions of sticks.

[maxglute]

Well game theory aside, the reality is if PRC weaponizes AI, there's a chance they may use it in the future. If US weaponizes AI, they'll definitely be using it to kill people within the calendar year. Employees have to factor that in, for PRC worker their killing people is hypothetical, for US worker, it's inevitable.

[khafra]

Do the same thing we did with the nuclear arms race: Treaties to limit and control it.

Obviously, we would have had more political leverage if our leaders had started working on a treaty before they crossed enough moral red lines to start a tech revolt, but we did not elect the sort of leaders that would do that.

[baq]

The obvious countries sign those treaties for political reasons. There are countries officially pretending they don’t have any nuclear warheads, but it is well known by everyone they have plenty and a capacity to deliver them. They sandbag for political, but also for religious reasons and that’s scary.

[nradov]

Treaties are pointless without a practical means of monitoring and enforcement. We're fortunate that nuclear weapons programs are difficult to hide from overhead imagery, seismographs, and fallout detectors. We can't verify what code is running in a Chinese data center or missile guidance system. It's just a ridiculous notion to equate AI or autonomous weapons with nukes in any way.

[andriy_koval]

treaties worked because both sides had large quantity of bombs. In this case, certain people do not want US to have AI bomb, while China and others will have it.

[dheera]

> American employees refuse to build this, but China's don't.

It's not American employees vs. China employees. No need to villainize China at every opportunity. Most Chinese employees are more similar to American employees than you think.

It's {top candidates who have their pick of employers} have the luxury to refuse to build this.

Mid-tier dude who can't land a job at any of the top AI companies and can code with Cursor and trying to pay their rent or medical bills will absolutely build AI for the military in return for having their rent paid.

This is regardless of whether it is in the US or China.

[nashashmi]

The reason it works is when you have less participants in an effort you have slower progress in that endeavor. Brilliant employees prohibiting their entire org to not support the development of bad things prevents less brilliant employees from doing bad things.

It is sort of like computers are amazing but can also be a privacy nightmare. Software engineers don’t help or coordinate with black hat hackers. So black hat hackers have a harder time refining their systems.

[impossiblefork]

Well, then military use of some US commercial AI systems will be subject to minimal restrictions while Chinese AI might not be.

Thus some people avoid having to see their work used for killing people or in mass surveillance, so that they're actually able to contribute to AI development instead of leaving the field.

[squibonpig]

As I understand anthropic refused two things: domestic surveillance in the US and weapons automated such that they could kill without a human in the loop. I don't think either of these would hamper the US against China in any meaningful way.

[baq]

That’s exactly why I think the principled position is naive in a tragedy of the commons situation we’re in - it isn’t a sci fi story with a happy ending, it’s the Manhattan project and 70+ years ago nazi and japanese data centers doing foundational model training would’ve been bombed to smithereens at any cost.

[AlecSchueler]

Are we comparing the PRC to early 40s Germany/Japan?

[nradov]

The PRC is comparable to early 30s Germany/Japan. We're on a dangerous course toward a devastating conflict if both sides can't reach a stable understanding.

[AlecSchueler]

> The PRC is comparable to early 30s Germany/Japan.

In what way?

[pjjpo]

Don't need to use China even, Microsoft, or Palantir, etc will continue to support the US military, likely using Google technology in the process (Guava, gRPC maybe?, k8s assuredly? etc).

Sorry but if you truly believe in technology not using in bad context, the only way to avoid it is to change careers. The issue with news like this is it's hard to actually trust the protesters, they probably are happy to clear their conscience personally while continuing to reap the benefits of living in the tech industry. Have your cake and eat it too.

Sometimes people do quit - they're probably the ones you want to hire if you care about ethics. Most don't though.

[jmyeet]

I'm going to give a shout out here to an episode of the excellent podcast Hardcore History, specifically Episode 59: The Destroyer of Worlds [1].

The development of the atomic bomb created a debate in American policy circles about how the US should react. Within a few years, the same debate occurred over developing thermonuclear weapons. The same question kept coming up: what if the enemy has these weapons and we don't?

Dan Carlin's position, which I happen to agree with, is that America chose wrong. It became both belligerent and paranoid to a degree that just wasn't the case before WW2. If you look up the history of regime changes at the hands of the US [2] then you can see it went into overdrive after 1945.

Part of the problem here I think is projection, the psychological phenomenon. It's also a cultural phenomenon. So, for example, when you have a historically oppressed people who are being potentially freed, the oppressors will fret that the formerly oppressed will rise up and kill them. This is projection.

We saw this exact thing play out with Emancipation. There was no mass revenge violence by the former slaves. If anything, there was more violence by the former oppressors against freed slaves and a system that excuded the violence (eg the Colfax massacre [3]).

I think nations can be guilty of this too. The US sees any other global power as a potential hegemonic, imperialist power that will dominate and exploit everyone around them because, well, that's what we do.

We also see this in how we view AI as a resource. We see it as something to be owned and gatekept such that some US company will become insanely wealthy further extracting every last dollar from every person on Earth.

So your comment belays a common fear that China will displace us as a global hegemonic, imperialist power despite there being zero evidence that China behaves in that fashion. American propaganda runs deep and the projection is strong so this will immediately cause some to say "but Tibet" or "but Taiwan" without really knowing anything any of those situations.

As just one example, the One China policy is the official policy of the US, the EU and almost every nation on Earth. "They might invade" I preemptively hear. They won't, partly because they can't but really because they don't need to. If the world already has the One China policy, why do anything? Oh and I said they can't because they can't. They don't have that military capability. If you think that, you don't know anything about war. Crossing 100 miles of ocean to invade an island with a army of over 500,000 is simply not possible.

Let me put it this way: the 17 or so miles of the English Channel stopped the German war machine despite having millions of soldiers.

Anyway, back to the point: this whole argument of "what if China does military AI?" is (IMHO) projection. If anything, China has shown that they won't allow a US tech company to control and gatekeep AI (eg by rreleasing DeepSeek). And if China gets AI, they're more than likely to use it to further raise people out of poverty and automate away more menial jobs without making those displaced workers homeless.

[1]: https://www.dancarlin.com/product/hardcore-history-59-the-de...

[2]: https://en.wikipedia.org/wiki/United_States_involvement_in_r...

[3]: https://en.wikipedia.org/wiki/Colfax_massacre

[Izikiel43]

> The US sees any other global power as a potential hegemonic, imperialist power that will dominate and exploit everyone around them because, well, that's what we do.

In the Cold War, this was the correct approach, the USSR was that.

[rudedogg]

> And if China gets AI, they're more than likely to use it to further raise people out of poverty and automate away more menial jobs without making those displaced workers homeless.

Your comment is very optimistic. But the quoted part reminded me of something I heard (again) about China using slave labor in their lithium mines:

https://www.state.gov/forced-labor-in-chinas-xinjiang-region...

[Hikikomori]

Actually great post. Can also recommend the behind the bastards episodes about this, maybe not as accurate as Carlin but more humourous.

https://podcasts.apple.com/se/podcast/part-one-the-men-who-m...

[platinumrad]

HN really can't handle comments like these huh.

[nradov]

You seem to be laboring under the naive belief that mainland China is a rational actor which will refrain from attacking Taiwan over fear of heavy losses and possible defeat. You might have been correct at some point, but that situation no longer obtains. Xi Jinping has successfully purged all potential rivals and personally taken over centralized control of all important decisions. We have no visibility into his thinking, so we have to assume the worst. If he orders the PLA to go then they'll go, regardless of consequences. Part of preparing for the eventuality involves building more effective autonomous weapons. There is no realistic alternative.

[jmyeet]

So I follow a number of China scholars and experts and I've yet to see any consensus about what these military purges actually mean.

It could be about corruption. You see this in the Russian military where paid-for tanks didn't exist because the generals had pocketed the money. It could be to have an expansionist policy. It could well be to not have an expansionist policy. The point is that nobody really knows yet.

But the string I really wanted to pull at was this idea that China isn't a "rational actor". It's lazy and really a thought-terminating cliche. It's certainly no basis for analysis or policy-making. It's kind of the final boss of justification. "Putin/Saddam/Xi/Castro/Maduro is crazy". That really just means you don't understand what's going on or want to ignore the facts.

We now have 50+ years (since really the end of the Cultural Revolution) of China acting in a very rational, very intentional and very long-term way. Xi's own history here is pretty interesting. He went from privileged child (his father was one of Mao's lieutenants) to being banished to working his way up through the party's ranks over decades.

It's a mistake (IMHO) to view Xi as a singular actor, let alone as a irrational autocrat. While the PRC and the CCP might be relatively new the systems and political structures can probably be traced back thousands of years. I'm thinking particularly of the bureaucratic reforms of the Qin Dynasty some ~2300 years ago.

What cannot be ignored is that a billion Chinese have seen a massive improvement in their living conditions during their lifetimes. Almost all of the people pulled out of extreme poverty in the 20th century were because of China (~800M). So although China is authoritarian, the government is extremely popular because of that increase in living conditions. It's something that we in the West have a hard time fathoming because our living conditions have been in decline since at least the 1970s.

[nradov]

Interesting but irrelevant. Hope is not a strategy. In intelligence analysis you have to look at capabilities and intents. We have no clear understanding of Xi's true intents, so for national security purposes we have to assume they're negative.

[CasualSuperman]

With current leadership, I think we're closer to Germany in this analogy.

[j16sdiz]

This is not answering the question.. and HN ain't US only.

You can say the same for any other country... What if Japan employee refuse, but American want that anyway? What if China employee refuse, but Russia employee want that anyway?

The implication are still the same -- social, culture, jurisdiction, national interest, company interest don't share the same boundary and don't align on their priorities.

[tokioyoyo]

I don’t think they’re refusing all military involvement. Autonomous-decision making is the problematic part.

[nradov]

The US military has deployed fully autonomous weapons systems since 1979. If you're worried about that then you're a little late.

[wewtyflakes]

It seems weird to equivocate the capabilities of "ai" in 1979 to what we have now; clearly it is on a different level.

[nradov]

Really? Autonomous weapons systems hold a lot of potential but as of today they haven't been very useful in real operations.

[tokioyoyo]

Yes, but we both know this is not the same kind of “autonomous weapons”.

[nradov]

No, I don't know that at all. The differences so far are only incremental. There is the potential for another revolution in military affairs due to autonomous systems but so far it hasn't actually arrived.

[b65e8bee43c2ed0]

my brother in Christ, what do you think the 40's America was like?

[ihsw]

This kind of inflammatory nonsense serves no purpose other than to be insulting and provocative.

[wewtyflakes]

I think it signals to other people that they should not feel alone in that "this is all **ed up". To that, I appreciated the comment.

[SpicyLemonZest]

Is there any reason to think that autonomous weapons are a critical strategic capability? It's hard to see what an unpiloted drone can do that a remotely piloted drone can't, other than perhaps human rights violations.

[remarkEon]

The simple version: Weapons systems are quickly advancing to the point where many of them can navigate and operate independent of human control. The obvious question here is at which point do we give these platforms release authority for lethal weapons. It becomes impractical to require (or even imagine, really) there to be a human "pilot" operating every single drone when you have hundreds or thousands of them operating in theater. That's really what this is about.

Think of it this way: mines installed in the seabed in wars past were "dumb", in that a passing ship had to happen into it. Imagine systems deployed underwater that were mobile, contained multiple torpedoes, and could strike warships with little to no warning given their small acoustic signature. It's the same principal as a mine (you leave it one spot, hope an enemy ship comes by), but the capabilities are far more advanced. If the system is not at least semi-autonomous than it might as well be a dumb mine again.

[boilerupnc]

Related: Reminds me of the slaughterbot video [0] and the Hated in the Nation Dark Mirror Episode [1][2]

[0] https://www.youtube.com/watch?v=9CO6M2HsoIA

[1] https://en.wikipedia.org/wiki/Hated_in_the_Nation

[2] https://www.youtube.com/watch?v=TN_kI7ELLJE

[nradov]

You're way behind. We gave those those platforms release authority for lethal weapons in 1979.

https://www.vp4association.com/aircraft-information-2/32-2/m...

[remarkEon]

I think scale is the differentiator here. Plus people get more nervous about things that fly than things that swim.

[nradov]

Remotely piloted drones can't operate at long ranges in a conflict against a near-peer adversary such as China. All of the high-bandwidth communications links will be degraded by a combination of jamming, cyber attacks, and anti-satellite weapons. Remote piloting will only be reliable using fiber optic cables (very short range) or direct line-of-sight transmission. So hardly practical in the Pacific theater of operations.

In an existential conflict no one cares about human rights. That's something for the winners to worry about after the shooting stops.

[boxedemp]

You don't need modern ai for that, it's been done decades ago.

Modern tools lend themselves more to information warfare and deobfuscation.

[andsoitis]

Faster decisions, less fatigue, etc.

[vasco]

It can turn around and bomb you.