[recursivecaveat]

In particular: *all the knowledge that AI has of nuclear weapons is freely available on the internet*. It's not superhuman, and there's no secret sauce data. If you just study the same PDFs and blog posts it has, you will acquire the same abilities. I cannot imagine anyone with the intent and immense financial and political resources to actually build a weapon would say that some study time is the only thing stopping them from detonating a nuke.

It is pretty convenient for the labs to frame the conversation around this though, since it is easy to address, very few paying customers are rejected, and sounds scary (so surely the less scary sounding stuff must be solved right?)

[derefr]

My hypothesis is that making the knowledge of how this stuff works accessible to the public results in a lot of false-positives (from people just playing around) that intelligence agencies have to then sift through / tune filters against; which creates a noise floor for real foreign nuke programs to hide in.

So governments ban anything that could result in false positives (since nobody needs to be doing any of that stuff outside of designated labs anyway), to lower that noise floor; to in turn make catching the foreign nuke programs tractable.

(It's a bit like how fancy mansions always have a completely flat and barren part of the property between an outer perimeter and the start of any gardens/outbuildings/water features/etc. That barren area is a killbox: since nothing is supposed to be there, anything at all that does appear there is a valid target for the manion's guards to shoot at [or otherwise engage with], without needing to get a clear identification and command approval first. This wouldn't work if the killbox was covered in vision-obscuring decorative features; nor if the mansion had employees, animals, etc. that had a valid reason to wander into the killbox. So such things are prevented, in order to make the problem of perimeter security tractable.)

[wongarsu]

But this knowledge is readily accessible today. At least for manhattan-project level bombs. For later developments you mostly get simplified overviews with important details left out. But even there you have communities speculating about this very publicly

The same is true for adjacent topics. Most LLMs will refuse to tell you how to make dynamite, youtube demonetises any videos about it, but it's right there in the wikipedia articles on dynamite and nitroglycerine

[bauldursdev]

I think they said increasing false positives because it would make it easier to generate at a mass scale. IDK the merits of the argument or what exactly they're saying would be done, but imagine pre-AI it might take someone quite a bit of manual effort to manufacture a plausible document regarding nuclear developments, but with AI it doesn't require so much work and is easier.

[derefr]

The main trick governments use isn't to hide the knowledge of how to build the stuff. It's rather to ban the sale of precursor chemicals and specialized devices (think: industrial-scale centrifuges) except through a government-observable KYC/AML-like chain-of-custody tracking scheme, that assumes/requires each intermediary and final consumer to be an organization certified as meeting certain security requirements.

Individuals obviously need not apply. But regular companies need not apply, either. Think "checkpoints and sign-out sheets that ensure that your own company will notice if some of this stuff disappears." Picture the sort of thing your mind might conjure if you've watched enough forensics protocol dramas and I say "evidence locker" and "tamper-evident seals" — except crossed with hazardous-materials handling policies.

The thing is, this whole chain-of-custody system can be pretty easily circumvented. I won't go deep into how (I'll just say: 1. there are principal-agent problems in academia, and 2. this system wasn't designed to handle sudden organizational bankruptcies well.) But the point is that a grey market for these precursor chemicals and specialized devices exists.

The main place that "false positive" events come from, that the state has to look into, is from people who manage to acquire precursor chemicals/devices without being part of any known chain of custody. (Which, note, doesn't mean that they did anything illegal per se. If it turns out they're just, say, a chemistry-education content creator, then the intelligence body just adds them to their knowledge graph and otherwise leaves them be. But they do have to do some interviewing to determine that first.)

To minimize the number of such events, the "knowledge" that is being truly suppressed here, isn't actually the knowledge of how to do the work; it's the knowledge of how to circumvent the chain-of-custody system. In other words: the logistics.

Information about "how to make a nuke" is general and evergreen; you can just absorb the lesson once and be good. So that info is just "out there", irrevocably. But information about "how to acquire the stuff to make a nuke" is both at least somewhat local to the country you're trying to do it from/in, and also changes all the time, as each state chases up and shuts down existing grey-market channels, and then new ones spring up to replace them. Thus, suppressing logistical knowledge is actually both useful and tractable. And so that's what states mostly go after.

(Mind you, the knowledge of "how to do the thing" does often end up roped into this knowledge-suppression scheme by overzealous downstream regulators who don't understand the load-bearing assumptions of the system they're working under.)

---

The worry states have about LLMs, I think, is that simply by scraping the web into a training dataset, they'll end up stumbling onto the right conversations (that sometimes do indeed happen anonymously in public) to end up with fresh + local chain-of-custody circumvention-logistics knowledge. (And it'd be very hard to "unpick" data like that from the training data.)

Or, even if they don't ingest the data at training time, they'll ingest "the places where that kind of info might end up", and thereby get so good at being "runtime demand-driven searching-and-scraping engines" for this type of thing that they'll be able to surface fresh sources of such info anyway — basically cranking the logistical-pipeline "reconnection speed" after state disruption of a supply channel down to near-zero.

Prohibiting the LLMs from speaking on this subject generally, prevents them specifically from enabling this specific fast-turnaround circumvention-logistics research use-case.

[rustcleaner]

Administrative convenience is no excuse to limit individual liberty, capacity, or knowledge. Individuals come before states!

[harrall]

Usually measures like these aren’t to stop the people with those kinds of deep resources.

With everything, there is a much bigger group of people in the middle that have “some resources” and “some desire” that these measures are surprisingly effective against.

Raise a $20 item by $1 and suddenly there’s fewer interested people, even though the cost difference is minor. Well, minor to some people but not to others.

But is limiting this information in an LLM the right move? Well that’s a different question.

[lazide]

The difficulty with creating nuclear weapons has been 99% in refining and processing the fuel, not the structure of them, for a very long time.

[HeatrayEnjoyer]

True for fission bombs. Less true for fusion bombs. The principal makeup and manufacturing of fusion device parts like tampers are still unknown to the public. Having a supply of HEU does not tell you how to assemble a functional triple stage device or how to utilize tritium, an isotope that measurably decreases in purity by the day.

[chasd00]

You need a fission bomb to ignite a fusion bomb btw.

[throwawayk7h]

That's rather meaningless. The scientists in the Manhattan project initially had less information than what is now available on the internet.

[aleph_minus_one]

> The scientists in the Manhattan project initially had less information than what is now available on the internet.

On the other hand: the Manhattan project had access to much better physicists than the typical terrorist group has. :-)

[throwaway85825]

But physicists today have much more information and compute and could be more productive.

[cultofmetatron]

its also hilarious when you consider that building nuclear weapons is fundamentally a supply chain problem. The taliban isn't going to suddenly have nuclear capabilities by asking chatgpt. Any adversarial nation that has the means to extract and concentrate fissile nuclera material probably has HUMAN scientists who spent years studying the problem in well funded labs.

[throwaway85825]

It's a way for AI labs to discuss safety while misdirecting from more mundane but widespread harms such as spam.