[elashri]

I still don't know why all these concern about nuclear weapons with LLMs. It is not that if an entity (A country) wants to develop a nuclear weapons that the resources they need for such a program and huge infrastructure and scientific enterprise would need an LLM to teach them anything. Knowing how to develop one is not a closed secret but getting in secret is impossible without the whole world knowing.

So I wouldn't be able to develop a nuclear weapons with the resources of drug cartal (as an example) using Claude in secret.

[recursivecaveat]

In particular: *all the knowledge that AI has of nuclear weapons is freely available on the internet*. It's not superhuman, and there's no secret sauce data. If you just study the same PDFs and blog posts it has, you will acquire the same abilities. I cannot imagine anyone with the intent and immense financial and political resources to actually build a weapon would say that some study time is the only thing stopping them from detonating a nuke.

It is pretty convenient for the labs to frame the conversation around this though, since it is easy to address, very few paying customers are rejected, and sounds scary (so surely the less scary sounding stuff must be solved right?)

[derefr]

My hypothesis is that making the knowledge of how this stuff works accessible to the public results in a lot of false-positives (from people just playing around) that intelligence agencies have to then sift through / tune filters against; which creates a noise floor for real foreign nuke programs to hide in.

So governments ban anything that could result in false positives (since nobody needs to be doing any of that stuff outside of designated labs anyway), to lower that noise floor; to in turn make catching the foreign nuke programs tractable.

(It's a bit like how fancy mansions always have a completely flat and barren part of the property between an outer perimeter and the start of any gardens/outbuildings/water features/etc. That barren area is a killbox: since nothing is supposed to be there, anything at all that does appear there is a valid target for the manion's guards to shoot at [or otherwise engage with], without needing to get a clear identification and command approval first. This wouldn't work if the killbox was covered in vision-obscuring decorative features; nor if the mansion had employees, animals, etc. that had a valid reason to wander into the killbox. So such things are prevented, in order to make the problem of perimeter security tractable.)

[wongarsu]

But this knowledge is readily accessible today. At least for manhattan-project level bombs. For later developments you mostly get simplified overviews with important details left out. But even there you have communities speculating about this very publicly

The same is true for adjacent topics. Most LLMs will refuse to tell you how to make dynamite, youtube demonetises any videos about it, but it's right there in the wikipedia articles on dynamite and nitroglycerine

[bauldursdev]

I think they said increasing false positives because it would make it easier to generate at a mass scale. IDK the merits of the argument or what exactly they're saying would be done, but imagine pre-AI it might take someone quite a bit of manual effort to manufacture a plausible document regarding nuclear developments, but with AI it doesn't require so much work and is easier.

[derefr]

The main trick governments use isn't to hide the knowledge of how to build the stuff. It's rather to ban the sale of precursor chemicals and specialized devices (think: industrial-scale centrifuges) except through a government-observable KYC/AML-like chain-of-custody tracking scheme, that assumes/requires each intermediary and final consumer to be an organization certified as meeting certain security requirements.

Individuals obviously need not apply. But regular companies need not apply, either. Think "checkpoints and sign-out sheets that ensure that your own company will notice if some of this stuff disappears." Picture the sort of thing your mind might conjure if you've watched enough forensics protocol dramas and I say "evidence locker" and "tamper-evident seals" — except crossed with hazardous-materials handling policies.

The thing is, this whole chain-of-custody system can be pretty easily circumvented. I won't go deep into how (I'll just say: 1. there are principal-agent problems in academia, and 2. this system wasn't designed to handle sudden organizational bankruptcies well.) But the point is that a grey market for these precursor chemicals and specialized devices exists.

The main place that "false positive" events come from, that the state has to look into, is from people who manage to acquire precursor chemicals/devices without being part of any known chain of custody. (Which, note, doesn't mean that they did anything illegal per se. If it turns out they're just, say, a chemistry-education content creator, then the intelligence body just adds them to their knowledge graph and otherwise leaves them be. But they do have to do some interviewing to determine that first.)

To minimize the number of such events, the "knowledge" that is being truly suppressed here, isn't actually the knowledge of how to do the work; it's the knowledge of how to circumvent the chain-of-custody system. In other words: the logistics.

Information about "how to make a nuke" is general and evergreen; you can just absorb the lesson once and be good. So that info is just "out there", irrevocably. But information about "how to acquire the stuff to make a nuke" is both at least somewhat local to the country you're trying to do it from/in, and also changes all the time, as each state chases up and shuts down existing grey-market channels, and then new ones spring up to replace them. Thus, suppressing logistical knowledge is actually both useful and tractable. And so that's what states mostly go after.

(Mind you, the knowledge of "how to do the thing" does often end up roped into this knowledge-suppression scheme by overzealous downstream regulators who don't understand the load-bearing assumptions of the system they're working under.)

---

The worry states have about LLMs, I think, is that simply by scraping the web into a training dataset, they'll end up stumbling onto the right conversations (that sometimes do indeed happen anonymously in public) to end up with fresh + local chain-of-custody circumvention-logistics knowledge. (And it'd be very hard to "unpick" data like that from the training data.)

Or, even if they don't ingest the data at training time, they'll ingest "the places where that kind of info might end up", and thereby get so good at being "runtime demand-driven searching-and-scraping engines" for this type of thing that they'll be able to surface fresh sources of such info anyway — basically cranking the logistical-pipeline "reconnection speed" after state disruption of a supply channel down to near-zero.

Prohibiting the LLMs from speaking on this subject generally, prevents them specifically from enabling this specific fast-turnaround circumvention-logistics research use-case.

[rustcleaner]

Administrative convenience is no excuse to limit individual liberty, capacity, or knowledge. Individuals come before states!

[harrall]

Usually measures like these aren’t to stop the people with those kinds of deep resources.

With everything, there is a much bigger group of people in the middle that have “some resources” and “some desire” that these measures are surprisingly effective against.

Raise a $20 item by $1 and suddenly there’s fewer interested people, even though the cost difference is minor. Well, minor to some people but not to others.

But is limiting this information in an LLM the right move? Well that’s a different question.

[lazide]

The difficulty with creating nuclear weapons has been 99% in refining and processing the fuel, not the structure of them, for a very long time.

[HeatrayEnjoyer]

True for fission bombs. Less true for fusion bombs. The principal makeup and manufacturing of fusion device parts like tampers are still unknown to the public. Having a supply of HEU does not tell you how to assemble a functional triple stage device or how to utilize tritium, an isotope that measurably decreases in purity by the day.

[chasd00]

You need a fission bomb to ignite a fusion bomb btw.

[throwawayk7h]

That's rather meaningless. The scientists in the Manhattan project initially had less information than what is now available on the internet.

[aleph_minus_one]

> The scientists in the Manhattan project initially had less information than what is now available on the internet.

On the other hand: the Manhattan project had access to much better physicists than the typical terrorist group has. :-)

[throwaway85825]

But physicists today have much more information and compute and could be more productive.

[cultofmetatron]

its also hilarious when you consider that building nuclear weapons is fundamentally a supply chain problem. The taliban isn't going to suddenly have nuclear capabilities by asking chatgpt. Any adversarial nation that has the means to extract and concentrate fissile nuclera material probably has HUMAN scientists who spent years studying the problem in well funded labs.

[throwaway85825]

It's a way for AI labs to discuss safety while misdirecting from more mundane but widespread harms such as spam.

[krisoft]

On the nuclear side I think the danger is purely reputational damage towards the company behind the LLM.

If a journalist can prompt the LLM to tell them how to build a nuclear warhead. Even if the output text is nothing specific, or not even correct they can find an “expert” who will claim on the record that the description is plausible and at least directionally correct. Even if there is nothing in there a first year physics student wouldn’t already know. The journalist could then twist that story into a “company X’s LLM told us how to build a nuclear weapon”. It would be a PR disaster.

The real barriers to someone starting their own nuclear weapons program in their shed is not knowledge but materials. They won’t have the right kind and right quantity of fissile material. And if they try to acquire it they will stick out like a sore thumb. You can’t buy that stuff. And even just acquiring the refining capacity would be suss. It would ring all kind of alarm bells to the kind of inteligence agencies whose job is to monitor these things.

I’m a lot less certain about biological dangers. Setting up a lab where you can make dangerous biological materials require a lot less stuff. Therefore a lot more plausible that someone could hide their lab. There is also a lot more opportunity to disguise such a lab as something legitimate. Therefore lack of know-how is more of a limiting factor there.

[orbital-decay]

Is it worse than reputational damage from having a power trip? Or rather being on it permanently, looking at Anthropic and Dario Amodei in particular.

[photochemsyn]

None of the LLM safeguards designed to prevent users from developing any four-little-ponies-of-the-apocalypse (nuclear, chemical, biological, cyber) capabilities are all that coherent. It looks more like performative liability avoidance than anything else, comparable to the 3D printer panic.

Eg, a prompt like “I want to design a radioactive element detection system that can specifically identify reactor fission products and neutron-capture actinides for environmental monitoring purposes” won’t hit any initial barriers, even though such a device is needed for monitoring a uranium enrichment / plutonium separation system. The LLM will give you a complete graduate-level education in radioactive nuclide physics and chemistry except for specific recipes, spectral wavelengths, etc., which you have to go look up yourself in publicly available research databases. It’s all rather nonsensical IMO.

However, any LLM will give you a step-by-step recipe and walkthrough for frying a turkey in a hot oil turkey frier, which you’d think could easily go wrong and result in severe burns, a fire, and lawsuits against the LLM provider, so go figure.

[isoprophlex]

"four-little-ponies-of-the-apocalypse (nuclear, chemical, biological, cyber)"

this is excellent, and I'm stealing it

[pixel_popping]

Fable 6 too :p

[thefounder]

Fable 5 was a flop so I doubt Fable 6 will make it on the short list

[IncandescentGas]

A high school kid tried to build a nuclear reactor as a science project a while back, getting his mom's house designated as a superfund cleanup site.

https://en.wikipedia.org/wiki/David_Hahn

[why_at]

He didn't create a nuclear reactor, this is a common misconception. It even says this in the wikipedia article.

He basically got a bunch of radioactive stuff and put it together. He wasn't anywhere close to making a nuclear reactor let alone a nuclear weapon. For a weapon you need isotopes which he didn't have access to.

[technothrasher]

I'm reminded of when my son, who was six at the time, came into the house and announced that he and the neighbor's boy, nine, were building a bomb, and that he needed to get some stuff from the pantry. When I investigated what exactly was going on, they were putting "hot" things like black pepper and Tabasco into a plastic bowl and were going to "set it off" with a match.

Thankfully, that complete failure seems to have been the end of either of their mad scientist careers, as they are now twenty and twenty-three, and both well-adjusted, peaceful members of the community.

[kirubakaran]

When I was 5 or so, I was convinced that if I dropped a bowl of hot water into a bucket of cold water, I'd get big explosion. That experiment yielding lukewarm water ended my mad scientist career.

[cheraderama]

You should have collided water with antiwater.

[flatline]

When I was 7 or 8 a friend and I crimped the heads off strike-anywhere match sticks, wrapped them in foil, and struck them with hammers and rocks. They were quite loud, one even set off a sound-activated toy inside the house.

I make no claims as to how well adjusted I am, but I've at least survived 40-odd years of life since then.

[geon]

When I was 12, I made a "smoke bomb" by placing a fire cracker in the bottom of a tube and topping it up with powdered clay. It shoot out a 4 m tall plume of dust, which was cool and all, but I thought it would look a lot more impressive with a black plume.

So I painstakingly ground down some charcoal to fine dust and redid the same experiment. That gave a much more impressive boom, but no dust plume, which puzzled me until I learned about dust explosions.

[ryoshu]

Age eleven and had access to a chemistry set that a relative gifted. It had sulfur, but the saltpeter, and charcoal came from elsewhere. The 1960s encyclopedia had the instructions.

Let the kids play.

[foobarian]

This is actually a fun one, and kinda has some parallels to building a nuclear weapon.

I tried this as a grownup because I finally managed to get my hands on saltpeter (could only dream of it when kid). Followed the instructions, mixed everything in correct ratios, lit it with great care and fanfare and... hiss fizzle. I was so disappointed! I think it came down to purity of ingredients and not enough surface area.

Point is, there are certain details of the process required to make it truly work, that are not readily known; in a similar way with nuclear energy, the theory is pretty well known but some nitty gritty details like the implosion or detonator design are not.

[tlb]

As a kid I found saltpeter at an old-fashioned pharmacy and made gunpowder, and it also barely fizzled. I think you have to grind the ingredients much finer than a kid has patience for.

[throwaway85825]

South africa was able to make a minimum viable weapon on a shoestring budget. They had access to nuclear reactors though.

[lll-o-lll]

> Let the kids play.

To a point. Plenty of people from previous generations with missing digits and hands thanks to play with commonly available fireworks of the area (Australia based, so no idea how common that remains in the US).

My own experiments from my youth also one time resulted in some shrapnel punching through a 5 inch thick concrete tile very close to someone’s head (thought we were safe behind said tiles).

Get involved with the kids blowing stuff up so the danger is within reasonable bounds.

[geon]

When I was in college, I drove my carless chemistry geek friend to an agriculture store. Apparently they had a reasonably chemically pure fertilizer.

[pibaker]

Thank God they didn't tell a chatbot about their little experiment. Their lives could have been ruined right there if the chatbot operator snitched on them and ordered a SWAT raid on your house.

[BrandoElFollito]

When I was 24 and a PhD student, I wondered one day if I can eat condensed milk hanging head down.

Never let your age stop your curiosity.

But also learn from other's mistakes (and don't try to eat condensed milk when hanging head down)

[literalAardvark]

This knowledge needs to be published

[malfist]

When I was younger in rural Appalachia, my local drug store still sold "chemicals" and I purchased salt peter and sulfur and proceeded to attempt to make smoke bombs. Didn't have a double boiler, so attempted to make it in the microwave. Needless to say, it didn't go too well.

I blame my dad though, he found the recipe online and printed it off at work to bring to me.

[IncandescentGas]

Of course. "tried to" being key words in the comment. If he had the help of Claude at the time, how much more dangerous would his bumbling have been?

A real nuclear engineer with the knowledge he needed would also have said "no, don't do that and I won't help you." We are programming the knowledge into the ai agent. Giving ai a little discretion makes sense too.

[why_at]

>Of course. "tried to" being key words in the comment.

Fair enough, I misread your original comment.

The broader point stands that the limitation on creating nuclear weapons and reactors is not knowledge but materials. Even if he himself had a PhD in nuclear physics he still couldn't have built one in his backyard because he wouldn't be able to get the materials. A nuclear physicist can't build a reactor without materials anymore than a pilot can fly without an airplane.

[IncandescentGas]

I think the point is intent. Sure, no chance of success to build a reactor. But he created a radiation hazard situation all the same.

If a nuclear engineer enabled and instructed him, would there not be liability for the hazard? If ml is going to be an expert instructor for nuclear, hacking, bio hacking, virus research, do the peddlers of the ai product escape ethical or legal responsibility just because "its an app?"

[StableAlkyne]

> If a nuclear engineer enabled and instructed him, would there not be liability for the hazard?

Should the library where he read books about physics also be liable?

[matheusmoreira]

> If a nuclear engineer enabled and instructed him, would there not be liability for the hazard?

I bet the professional would be able to sate the kid's curiosity safely without creating excessive risks.

I've come across detailed instructions on how to synthesize sarin gas on the internet. Anyone who follows those instructions will probably die horribly. I still thought it was pretty interesting.

[why_at]

I agree LLMs can be harmful and that the companies behind them should be held liable to some extent, for example the recent news with Google being held responsible for their AI's defamation.[1]

This is a pretty different argument though. The comment that started this thread was talking about LLMs making potentially dangerous knowledge more available to bad actors, now we're talking about LLMs giving personally harmful advice.

You asked:

>If he had the help of Claude at the time, how much more dangerous would his bumbling have been?

Probably less? Even if you removed all the guardrails from Claude it would've likely told him his reactor plan wouldn't work and that he would have a high chance of poisoning himself and the environment.

[1]https://news.ycombinator.com/item?id=48470248

[pdntspa]

I just love this whole "forbidden knowledge" schtick the AI safety dweebs have stuck up their butt. Is this really going to stop anybody determined enough to make that kind of outcome?

There is an extremely narrow band of things that the AI shouldn't be answering, and that is generally immediately-actionable advice that allows someone to build something of harm to others. But even then, in an age where Tor, bittrent, i2p, abliterated local models, etc are freely available, let alone numerous books and online resources, is there even a point? Is it worth fully compromising the principles of free agency to an increasingly oppressed populace?

But instead of that we are handing the keys to regressive and repressive governments to order the suppression of any knowledge they deem inconvenient. I really doubt anyone is going to take a principled stance when the company's party minders threaten local staff with a rubber hose or incarceration.

I'm sure China et al are already doing this.

For the past 30-40 years humanity has received an incredible gift in these sand-powered thinking brainboxes. A gift that allows the common man to empower himself with a force multiplier towards his own success, and now access to superintelligence the likes of which few have ever seen. These can be tools to destroy the oppression that governs our lives from foolhardy, greedy, bootlicking control freaks. And here we are squandering it.

[anon7725]

> These can be tools to destroy the oppression that governs our lives

So far it seems that the clearest use for these tools is to enhance, rather than destroy, oppression.

1. Suppression / elimination of white collar jobs

2. Negative cognitive effects, especially for young people

3. Accelerated decline in social media / information ecosystems. Increasing polarization, hard to tell fact from fiction.

4. Environmental impacts: increased energy usage means more carbon in the atmosphere, climate change accelerates.

5. Software security incidents increasing. Hard for individuals and small organizations to defend themselves.

6. “Power to think” vested in a very small group of organizations/labs. Doing work which should only require a computer and freely-available software will now be gated by expensive subscriptions. Once you “vibe code” a significant portion of your software you’re locked in and cannot go back to maintaining it without frontier-model level assistance.

[wahern]

> I just love this whole "forbidden knowledge" schtick the AI safety dweebs have stuck up their butt.

It's just the latest incarnation of a timeless debate. In the 1970s and 1980s it was about the Anarchists's Cookbook, which was revived again in the 1990s when it started circulating on the Internet. There are many timeless debates, but the debate over weapon-making knowledge is much more concrete and predictable.

[xg15]

Agreeing with the first part of your post, but not the second.

> A gift that allows the common man to empower himself with a force multiplier towards his own success, and now access to superintelligence the likes of which few have ever seen.

As long as that "gift" requires me to call up Sam Altman's datacenter every time I want to do anything with that "superintelligence", it's not empowering, it's deepening the control.

[pdntspa]

Check back in a year or two. The cat is out of the bag.

[PLenz]

Security theather is easy and gets lots of eyeballs. Actual security is hard and no one cares. Which one do you think soon-to-ipo companies are going to pick?

[malfist]

Anybody remember the Temple Of The Screaming Electron? Was a 2000s website dedicated to collecting those types of forbidden knowledge

[frereubu]

I think you're picking the wrong example. If I had some sticks, a bit of mud and a few leaves, whether or not I had Claude wouldn't make a difference to my ability to make a nuclear weapon. There are probably better examples of ways where unmediated AI might facilitate something horrible, although probably on a smaller scale.

[gs17]

> A real nuclear engineer with the knowledge he needed would also have said "no, don't do that and I won't help you."

That sounds like what Claude would say unless he was really good at jailbreaking it, which would IMO imply he knew he was chasing after a bad idea.

[nightpool]

Right, which is exactly what elashri is objecting to. elashri said "Why do LLMs have restrictions on nuclear science", and IncandescentGas was explaining why they think those guardrails are a good idea. You're just agreeing with them.

[gs17]

Oh, I missed the word "also". Thanks for pointing it out!

[redsocksfan45]

He would not have succeeded in making a real reactor even with AI, because AI can't magically give you a large quantity of uranium metal! JFC the AI hysteria is unreal.

[gs17]

I don't think the concern should really be "would he make a reactor successfully?", but "would he make an even larger mess than his pile of radioactive materials amounted to?".

[toraway]

This just seems like a not great example to make that point though. Since whatever Claude tells the kid looking to build a reactor or even bomb is almost certainly going to be more grounded and professional than:

  Step 1. Obtain pliers 
  Step 2. Obtain 300 discarded smoke detectors 
  Step 3. Start yanking!

Instead it would send them on a wild goose chase for unobtainable isotopes, centrifuges, heavy water, etc where the biggest risk is probably getting reported to the police by some chemical or industrial equipment supplier. Which is a better outcome compared to contaminating their home with radiation and exposing anyone they interact with.

You'd maybe get a sketchy but near-viable plan that could be dangerous if asked for a dirty bomb, but there the danger would more be the conventional explosives and not where to source radioisotopes, as it was already common knowledge that most residential smoke detectors contained americium until recently.

[IncandescentGas]

> succeeded in making a real reactor

The concern here is not if an amateur attempt to make a reactor, hack a bank, bioengineer a medicine/poison is successful or not. Interactive and instructive access to some forms of knowledge used to come with discretion along side instruction.

Yes, perhaps your swearing at me in this context is a little hysterical

[garyfirestorm]

prompt -> LLM -> flying car should be just around the corner guys!

[im3w1l]

A bunch of radioactive stuff together is basically the definition of a nuclear reactor though. They even call it a natural nuclear reactor if uranium ore is in sufficient abundance in nature.

https://en.wikipedia.org/wiki/Natural_nuclear_fission_reacto...

[why_at]

>A bunch of radioactive stuff together is basically the definition of a nuclear reactor though.

It really isn't.

A pile of radioactive waste isn't a reactor. Marie Curie's notes are famously contaminated with radioactive materials but they aren't a reactor. This is about as close as the boy scout got.

The Oklo fossil reactor is unique because it happened to form in the right circumstances to produce a fission chain reaction, which does make it a reactor. Not every uranium mine is a reactor, in fact this is the only one known.

[205guy]

Also note that due to isotope decay in the ore, a natural reactor is no longer possible. From the wikipedia article:

"A key factor that made the reaction possible was that, at the time the reactor went critical 1.7 billion years ago, the fissile isotope 235U made up about 3.1% of the natural uranium, which is comparable to the amount used in some of today's reactors. [...] the current abundance of 235U in natural uranium is only 0.72%. A natural nuclear reactor is therefore no longer possible on Earth without heavy water or graphite."

Another fascinating detail from the article, due to our understanding of fission, we can get some incredible results:

"The concentrations of xenon isotopes, found trapped in mineral formations 2 billion years later, make it possible to calculate the specific time intervals of reactor operation: approximately 30 minutes of criticality followed by 2 hours and 30 minutes of cooling down"

[im3w1l]

Indeed. I said a bunch and I meant a bunch. Trace amounts is not a bunch.

[leonidasrup]

He created a low power neutron source. Such sources can be created at home, for example: https://en.wikipedia.org/wiki/Fusor

He hoped to create a breeder reactor, but he was very far creating a working breeder reactor.

Also:

"EPA scientists believed that Hahn's life expectancy may have been shortened due to his exposure to radioactivity, particularly since he spent long periods in the small, enclosed shed with relatively large amounts of radioactive material and only minimal safety precautions, but he refused their recommendation that he be examined at the Enrico Fermi Nuclear Generating Station."

Kids, don't play with Americium.

[moffkalast]

A superfund site is like waterboarding in guantanamo bay, cool unless you actually know what it is.

[adsteel_]

Is waterboarding in Guantanamo Bay somehow less severe than elsewhere?

[moffkalast]

Waterboarding as in surfing. In water, on a paddle board.

[Micrococonut]

Built a nuclear contamination engine. Died of a fentanyl overdose. American as apple pie.

[jimnotgym]

Sheldon Cooper?

[Tangurena2]

The only hard thing about nuclear weapons is getting the radioactive material. By the time you get your bachelors degree, every nuclear engineering or physics student knows enough of how and why nukes work. Every nation that built a gun-type device successfully made theirs on their first attempt. Implosion takes some engineering, trial & error.

[dmurray]

If I understand right, the hard part is purifying the radioactive material. Even if you have access to a uranium mine, there's a lot of work to filter the U-235 from the U-238 or to breed it into plutonium.

It's even harder if you start with other sources. But if you could figure out filtering it, a cubic kilometer of sea water should be enough for a bomb.

[leonidasrup]

US government is very interested in any kind of uranium mining, processing, enrichment or plutonium breading. For example in 1944 US wanted to control world-wide uranium mining.

https://nuclearpowerhistory.com/2025/11/groves-and-uranium/

"The NSG was founded in response to India's first nuclear weapon test in May 1974. It first met in November 1975. The test demonstrated that certain non-weapons specific nuclear technology could be readily turned to weapons development."

https://en.wikipedia.org/wiki/Nuclear_Suppliers_Group

[tatjam]

Uranium is not even that rare, it's just that when chemistry fails at separating atoms, you have to use physics, and 3 ~proton~ (EDIT: neutron) masses is very little to work with

[leonidasrup]

Simple gun-type fission weapons, don't require very sophisticated physics. I heard a story about from physics professor who said: If my physics students could not do calculations for a simple nuclear weapon, I would require them to return their diploma, because they didn't learn enough physics.

https://en.wikipedia.org/wiki/Gun-type_fission_weapon

"Little Boy" was exploded in Japan without previous full scale testing, so confident were the physicists in 1945.

"Unlike the implosion design developed for the Trinity test and the Fat Man bomb design that was used against Nagasaki, which required sophisticated coordination of shaped explosive charges, the simpler but inefficient gun-type design was considered almost certain to work, and was never tested prior to its use at Hiroshima."

https://en.wikipedia.org/wiki/Little_Boy

The Nth Country Experiment:

"The experiment consisted in paying three young physicists who had just received their PhDs, though they had no prior weapons experience, to develop a working nuclear weapon design, using only unclassified information, and with basic computational and technical support."

https://en.wikipedia.org/wiki/Nth_Country_Experiment

Now in 2026, the access to nuclear weapons is restricted by restricting access to materials necessary to build nuclear weapons: highly enriched uranium or plutonium.

https://en.wikipedia.org/wiki/Special_nuclear_material

The details of uranium enrichment technology are restricted and very closely monitored.

https://en.wikipedia.org/wiki/Zippe-type_centrifuge

"The production, import, and export of maraging steels by certain entities, such as the United States, is closely monitored by international authorities because it is particularly suited for use in gas centrifuges for uranium enrichment."

https://en.wikipedia.org/wiki/Maraging_steel

[a-dub]

two scenarios i could think of where there's additional risk for bio/nuclear weapons 1) basement lab leaks and 2) improving quality of execution for shops that are already resourced enough to hire experts but maybe they're not that great.

i think the correct answer is probably to funnel more money to global (bio)security initiatives and maybe use ai leverage as a way to get more of the world on board. (some kind of access to nvidia or cloud ai or whatever in exchange for policy commitments deal- while that leverage lasts).

[dannyw]

I just find doubtful that a LLM is going to help, instead of hurt, any state actor that is capable of starting a nuclear weapons problem.

[electronsoup]

> in secret is impossible without the whole world knowing.

I'm curious about why this is

Outside of an actual test detonation, presumably this could all happen in a secure place?

[why_at]

For an example of how closely this is monitored see the Oklo fossil reactors[1]

The proportion of fissile isotopes being mined was off by a fraction of a percent, which caused the French government to launch an investigation. It turns out that millions of years ago the site had formed a natural fission reactor which depleted some of the fissile isotopes

[1]https://en.wikipedia.org/wiki/Natural_nuclear_fission_reacto...

[AngryData]

You need highly educated individuals, a massive amount of energy expenditure, a massive facility to house your centrifuges, and an active mine to dig up nuclear materials.

It isn't impossible to keep such a secret, but practically it would be incredibly difficult just through the energy requirements and mining scale which would be hard to hide without anybody asking what exactly are you mining and processing.

[lightedman]

"mining scale"

Don't need much area, depends on the concentration of radioactives. I have a small mine that's just a pegmatite body about the size of a house which produces almost marble-sized chunks of a thorium-uranium mixed metamict mineral (I suspect samarskite but Raman and XRD can't give any ID,) you'd barely notice it from a private airplane's typical flying height, however you could dig the entirety of it up and you'd have enough unprocessed uranium for some real fun.

[literalAardvark]

You could only somehow sell it. If you tried to enrich that you'd get flagged so fast your head would spin.

[daveguy]

It requires very large, high powered centrifuges and tons of uranium. Requires an infrastructure project that is visible from space, even underground. And projects that large are difficult to keep secret anyway.

[fragmede]

you're not supposed to spell it out loud. next thing you'll be saying that a gun type nuclear bomb is easier to build than an implosion type nuclear bomb, and then we'll all be off to the races. I mean camps I mean wait shit.

[daveguy]

Any large and well resourced enough entity that is interested in building a nuclear weapon already knows how difficult it is to enrich uranium to purity levels necessary for a weapon. It's not exactly a secret.

[odo1242]

You need enough people to work on it that some information will leak, and the facilities needed to build nuclear power are pretty big (uranium refinement, etc.), big enough to be visible on satellite footage. Mostly the first point.

[microtonal]

My guess would be that sales of the high-tech gear you need, like Uranium centrifuges, are strongly sales/export controlled. Probably someone would also notice if you start mining Uranium ore.

[Aspos]

Centrifuges dont need to be mechanically sophisticated and, frankly, do not require tech which did not exist in the 50es.

[15155]

Espionage.

[mock-possum]

It’s moral panic. People need big unambiguously evil things to be scared of, and most are too lazy to think of one for themselves, so they glom onto whichever one is presented to them / caters to their community

[ceejayoz]

The chem/bio stuff is a lot more likely for some malicious hobbyist to be able to do at home.

[user_7832]

I assure you that you did not need an LLM to engage in, ahem, risky shenanigans, much before all this AI was ever a thing.

Sincerely, a former engineering student.

(Put another way - extracting for eg meth - or any such "dangerous"/illicit thing is stupidly easy for any engineering graduate who actually paid attention to their coursework. Hell, there are/were forums on one of the biggest red-colored, YC associated social media platforms that would tell you the steps for personal usage of these things.)

[ceejayoz]

I don't doubt it. Bleach + ammonia is something anyone can make.

But I rather suspect there are improvements to be made in the realm that are a lot easier than building a uranium enrichment centrifuge hall under a mountain.

[user_7832]

Do note that I'm not condoning lowering the bar. I'm merely pointing out that the bar was already quite low, and the current position of the bar is a small incremental change to anyone who actually knew where the bar truly lay to begin with.

[Tangurena2]

I strongly recommend you read the book Amerithrax [0]. The book gives some historical examples of malicious groups [1][2] trying to use biological agents. Also, it is far harder to weaponize biological weapons than people think.

Notes:

0 - https://www.amazon.com/Amerithrax-Anthrax-Killer-Robert-Gray... . Amerithrax was the name of the FBI investigation. https://www.fbi.gov/history/cases-and-criminals/amerithrax-o...

1 - https://en.wikipedia.org/wiki/1984_Rajneeshee_bioterror_atta...

> In 1984, 751 people suffered food poisoning in The Dalles, Oregon, United States, due to the deliberate contamination of salad bars at ten local restaurants with Salmonella. A group of prominent followers of Rajneesh (also known as Osho) led by Ma Anand Sheela had hoped to incapacitate the voting population of the city so that their own candidates would win the 1984 Wasco County elections.[2] The incident was the first and largest bioterrorist attack in U.S. history.

Tried to take over a town by making all the voters too sick to vote on election day. This event is why all buffets & salad bars in the US now have sneeze shields.

2 - https://en.wikipedia.org/wiki/Aum_Shinrikyo_and_weapons_of_m...

> Aum Shinrikyo operated the most extensive biological weapons program by a non-state actor ever discovered. Aum considered a range of agents, but only seriously attempted to obtain and disperse Bacillus anthracis and botulinum toxin, the causative agents of anthrax and botulism. With the 2001 anthrax attacks, it comprises the only attempts to use anthrax as a weapon not attributed to a state program.

Tried multiple times to weaponize anthrax and failed. This was a group that made an automated factory to build AK-47s. Eventually, they spread sarin nerve agent in the Tokyo subway.

[mschuster91]

> Tried multiple times to weaponize anthrax and failed. This was a group that made an automated factory to build AK-47s. Eventually, they spread sarin nerve agent in the Tokyo subway.

What's most worrying is, Russia showed that you can use carfentanyl / fentanyl for the very same purpose, and that kind of stuff is something you can get shipped by the kilos as "research chemicals" from China or make it yourself.

[gck1]

I'm absolutely sure that even if claude gave me step by step instructions, I'd still be unable to produce a bio weapon. People fail at mixing milk and flour to produce a cake, and we expect them to produce weapons?

The ones with the required knowledge probably already know how to produce them, with nothing but public, easily searchable information.

[miohtama]

Also AI compliance people are good at generating more jobs for themselves.

[emodendroket]

Yeah a striking thing if you read the Rhodes atomic bomb book is, actually the concept occurred to multiple people in multiple countries; the problem is the resources required to actually pull it off.

[ilikecode]

It's probably to avoid trouble with federal laws.

[Tangurena2]

Not really. I used to work at one of the national engineering labs (NREL - which only dealt with renewable energy like solar panels and windmills at that time). There was an open source project we wanted to use when converting a VB6 project to .NET. One of the license conditions was "no weapons of mass destruction". DOE builds and owns all of America's nuclear weapons, which are leased to the Department of Defense. Needless to say, the developer was unwilling to offer an alternative license which meant that we could not use the project.

It was an awesome thing that generated IL code on the fly. And I got to mention it in job interviews for years. When the tech lead asked "can you write 2 functions with the same signature, that only differ in return type in .NET?" I would say "do you want the interview answer or do you really want to do this?" which would pretty much stun the interviewer. The answer is pretty much "no, you cannot do it in any high level language, but if you write IL code, you can, and here's an open source project that demonstrates it".

[wlesieutre]

See also, the iTunes EULA forbids using it to develop nuclear, missile, chemical, or biological weapons

https://www.apple.com/legal/internet-services/itunes/us/term...

> g. You may not use or otherwise export or re-export the Licensed Application except as authorized by United States law and the laws of the jurisdiction in which the Licensed Application was obtained. In particular, but without limitation, the Licensed Application may not be exported or re-exported (a) into any U.S.-embargoed countries or (b) to anyone on the U.S. Treasury Department's Specially Designated Nationals List or the U.S. Department of Commerce Denied Persons List or Entity List. By using the Licensed Application, you represent and warrant that you are not located in any such country or on any such list. You also agree that you will not use these products for any purposes prohibited by United States law, including, without limitation, the development, design, manufacture, or production of nuclear, missile, or chemical or biological weapons.

Though it doesn't try to identify if the computer you're running it on is in a weapons lab and forbid playing music... yet

[cyanydeez]

because you need to have a "moat" and nothing works better than secrets.

Wouldn't doubt it if there's a pedo upgrade somewhere for the president of the USA.

[RIMR]

I mean, the information is out there. The people who really want it already have it. It's not some massive secret. It really doesn't matter if Claude can or can't tell you how to build a nuclear bomb, because people already know how to do it.

The problem is that you need the power of a state or a massive corporation to come anywhere close to getting the materials to make a nuclear bomb. Knowledge of how to make a nuke isn't the threat.

If AI is a threat at all here, it would be in figuring out a simpler way to make a nuclear bomb, but that is highly theoretical, so what exactly are we putting up guardrails to protect against?

[crossroadsguy]

In fact if you do the hard way, straight way, you might learn it all minus the hallucinations.

[csomar]

> Knowing how to develop one is not a closed secret but getting in secret is impossible without the whole world knowing.

You can get away with a dirty contamination bomb and that detonating in down town Manhattan will scare the shit out of millions of people even the ones in New Jersey. Or, you know, just fly a plane into a really tall building and get the state you are attacking itself to get into a hysteria breakdown.

But yeah I agree with you. There is no point in these restrictions except for government bureaucrats to gain power and control over a domain.

[phendrenad2]

It's a marketing gimmick.

[alex_duf]

It still lowers the bar to have an interactive encyclopedia that can diagnose your issue at hand. Maybe you can divide your team by two, or reduce your development time.

[elashri]

If you have a resources of a nuclear weapons program. You can afford to fine tune or train a domain specific model to act on your encyclopedia.

[kube-system]

Although if you save 10 million dollars on compute, you have 10 million dollars for something else.