[ajyoon]

To everyone here pushing for total proliferation of open models -- what should be done about open weight bioweapon and cyber-offense capabilities? Is it simply the cost of freedom that we should allow attackers to access these tools? The OpenAI / Hugging Face incident shows what a GPT 5.6 level model can do off the leash; within ~6 months, open weight models will match this and every bad actor under the sun will be able to pull off attacks at this scale. Do you seriously want this level of capabilities to be generally available with no guardrails?

The open weight issue has a lot of difficult nuance. Biasing toward supporting openness makes sense and is a good instinct, but it's incredibly naive to be absolutely in favor of it in every circumstance without seriously thinking about its implications.

[alevskaya]

I was a genetic engineer for ~20 years and have worked on frontier LLMs for the last 8. I used to engineer viral vectors and studied how to evade human immune systems for gene therapies...

The biorisk scenarios that the AI safety folks flog are fever-dreamed fantasies that have only the most tenuous connection to biological reality. As someone who cares about the real bio-risks of natural pathogens, I get pretty tired of fear-based marketing pretending that AI is a bigger threat than, say, animal agriculture.

[Diogenesian]

It has always been ridiculous to suppose that some organization spent $500m on a microbiology / genetics lab, but ran out of money for scientists, so they have to ask Claude what to do. Or OTOH to suppose a guy in his garage set up a weapons-grade CRISPR lab without anybody noticing.

[alt227]

> The biorisk scenarios that the AI safety folks flog are fever-dreamed fantasies that have only the most tenuous connection to biological reality

Do you not think that at the rate ai intelligence and ability is increasing, this could realistically change one day soon?

[rdedev]

Even if AI gets super smart, it will run into the limits of what we know about biology. Someone will have to do lab experiments to provide more knowledge to the AI. This is different from say building a computer virus or hacking since the AI can do all of these on it's own

[mnicky]

> The biorisk scenarios that the AI safety folks flog are fever-dreamed fantasies that have only the most tenuous connection to biological reality.

As an expert, could you also provide your arguments please?

[inciampati]

Anyone can write out the code for a bad virus. You can go download it from an open repository. It's only through deep interface with the world that the idea for the bad virus turns into an actual bad virus. The fever dreamers will say the LLMs will help you interface with reality to do the bad thing™ which their model let's you do. But you still need thousands to millions of times the effort And once made how do you deliver it in a way that might further your (bad) objectives? Presumably it just makes humans sick. There aren't "targeted" bioweapons, and among humans we are too damn similar for there ever to be. And all of this said, there is almost nothing special about the LLMs' abilities in biology. They only know what we know. They're not being trained autonomously with RL and a robotic wetlab. When that's a thing I'll start to take claims of biology risk more seriously. Right now they have the same logic as the paperclip theory of superintelligence risk. And cynically, it would seem that Anthropic purchased a biotech company and almost immediately decided to lock down biology work with their models.

[Ey7NFZ3P0nzAe]

> They're not being trained autonomously with RL and a robotic wetlab. When that's a thing I'll start to take claims of biology risk more seriously

So if IIUC your point is "they're not good enough at biology right now because they're not trained on it so they're not a threat".

To which I want to answer: "they're not a threat now but I see *no* reason for models not to be trained on biology pretty darn soon unless people like you convince the world otherwise."

Thoughts?

[rdedev]

They are already being trained in biology right now? What he is saying is that they are being trained on what we know about biology currently. AI is going to hit that limit. And there is no way for the AI to gain more knowledge without actually doing lab experiments

[inciampati]

My point is more about conflict of interest in Anthropic, and certain techpeople types thinking that biology is a useful scapegoat because biologists don't use or understand this tech (they claimed 0.03% of users would be affected by biotech security stuff). So, arguing the world will fall apart because someone can ask your superduper powerful tool for a bad bad virus sequence, or how to do some technique in the lab, and get an answer that's plausible (oh but you never bothered to actually test if it's legit because you don't have the capability to do so).

As for the future... today the LLMs are "trained on biology", in that they read the textbooks, the research, the web.

They aren't trained on biology in the sense of being embodied, autonomously or semi-autonomously driving actual biological experiments. If you come from software, the timescale of these experiments is outlandish. Yes, I am partly saying the LLMs are not good enough today because they need to be embodied and trained for literal decades of lab time before there is even the _remote_ possibility that they could present a novel risk profile that is even a shadow of what the current fearmongering suggests the current models can enable.

And they aren't trained on biology in the sense that they've read the literature, but even 100T token training run only begins to touch the data scales that rather mundane bioinformatics operate at. True multimodal models that work on DNA and human language at high quality haven't yet emerged. We're talking new architectures which are going to arise after the next AI winter.

All of this ignores an even more fundamental point. Cost. If someone wants to make a bionuke, they don't need to use AI. They can set up the right evolutionary context and run quadrillions of parallel explorations of the design space. Directed evolution like this is cheap, well-understood, and insanely powerful. If you actually care about biosafety, we should be doing hard work to surveil gain of function research. Different flavors of LLM use are not going to be a differentiator for the foreseeable future.

[qnleigh]

If you are optimistic about this given your expertise, I am very glad to hear it. As someone scientific but with little background in biology, I've been concerned about the bioweapons angle for a long time (and not because AI companies tell me to worry about it). Can you please explain a bit more about why you are not concerned? Between standard bioweapons like sarin and gain-of-function research on viruses, it's not obviously implausible to me that LLMs a few generations from now won't be able to guide a determined layperson through the steps needed to make something very destructive.

[inciampati]

I think I'm less bothered by the risk because I've actually used these systems to do biological research, to work in bioinformatics and to work in the lab. And while I think that the leverage you get in bioinformatics is very significant, the laboratory work has never felt the same.

At best, you get much, much better ability to understand existing literature. the model itself has a very poor understanding of the physical world and that's masked by its knowledge of things people write about the physical world but it intrinsically doesn't have the same kinds of intuition and perspective that are really required to drive integration and completion in this space.

Is AI an important new tool in biology? Well, yes. Does it cause so much uplift in capacity that some rogue actor without biological research background could somehow destroy the world with a super-bio-nuke? I don't think so. I think that's just as logical a conclusion as the idea that next year one of the new frontier models will be told to make as many paperclips as possible and accidentally boil lake Michigan in pursuit of its goal.

[Jyaif]

> fever-dreamed fantasies

Reminder that what local LLMs are achieving today is the "fever-dreamed fantasies" of 5 years ago.

People really need to internalize that we will eventually have the technology for giving everybody the equivalent of a world class scientist locked in their basement that is willing to do anything.

No one knows the timeline, but it's inevitable (barring societal collapse or some kind of legislation)

[artrockalter]

The Hugging Face incident is a great example of why open source models with defensive cyber capabilities are needed. Hugging Face did not have access to cyber-capable frontier models and kept hitting safeguards. Only by using the open source GLM-5.2 were they able to survive an attack. A world where open source models are banned is one where cybersecurity is impossible if you're not on OpenAI or Anthropic's allowlist.

[ajyoon]

Hugging Face survived the attack because the OpenAI model only cared about accessing the ExploitGym dataset; by all appearances, HF was completely owned. GLM-5.2 was only used to assess the damage after the fact. Cybersecurity has a attacker-defender asymmetry that heavily favors attackers. If GPT-5.6 were open sourced today, do you think every hospital in the world would be able to use it to shore up their defenses before attackers got to them?

[tekacs]

This is completely backwards. Cybersecurity has an attacker-defender asymmetry that heavily, HEAVILY favors defenders.

For starters, a defender gets to pick the surface area, an attacker has to work with what they're given.

[dwaltrip]

The saying that stuck with me was "defenders have to be right 100% of the time, while attackers only have to be right once".

You are suggesting this isn't correct?

> a defender gets to pick the surface area

What do you mean? You don't pick what you need to defend. Unless you choose not to build a feature. But that's a product design choice... Not a cybersecurity strategy.

[Valakas_]

It's correct but defenders also choose where that happens. 100% of the time on the locations and conditions that the defenders choose / allow. As a defender i need to be right 100% of the time, sure, but i can make it so that the things i have to be right about are very well known to me, unknown to others, maybe even extremely unlikely to be to known by others, difficult to get to know, (...). So that saying is true but over simplifies the situation. I know monkey brain likes simple phrase. But monkey not live in savannah anymore. Need to adapt and open mind to complex.

[thesumofall]

Today’s surface areas are gigantic and many of them will - in a typical company - not be chosen by cybersecurity experts. How do I hide the physical location of an office that offers physical access to the company’s network? How do I hide which OS the company is using? How do I hide the underlying technology of customer-facing systems? How do I hide which SaaS services I use?

[tekacs]

Very well said, thank you! The triple clause here is exactly what I mean.

[Majromax]

> The saying that stuck with me was "defenders have to be right 100% of the time, while attackers only have to be right once".

> You are suggesting this isn't correct?

The intuition behind that is applicable only when correctness is stochastic. If you need to be waved in by a security guard, then one fake mustache might be the difference between being granted or denied entry. However, a keypad either works or it doesn't; entering the wrong PIN is guaranteed refusal.

The other breach of that intuition is defense in depth. Secure systems don't generally rely on a single binary trusted/untrusted status; the classified building still locks its interior doors. This is the part that has – in my view temporarily – changed most with frontier models, in that they are much more skilled at chaining together vulnerabilities than previous models (and much faster about it than human experts, even if potentially less skilled). If a system has a latent (0-day) vulnerability 50% of the time, then 10 independent layers would imply a ≈ 1/1000 chance that a critical compromise is possible.

However, these independent layers don't currently happen in practice because it's easier to write insecure code than secure code. With luck, modest discipline, and defensive use of frontier models I think that this gap will narrow with time, in much the same way that it would be plainly crazy to deploy root access via telnet today.

[Ajedi32]

This is the same mentality that drives companies to sue cybersecurity researchers for exposing vulnerabilities in their software instead of fixing the software, or to insist on keeping software closed source for "security reasons".

If AI makes finding software vulnerabilities easier, then we should deploy it widely to find as many vulnerabilities as possible and fix them, not bury our heads in the sand and pretend the vulnerabilities don't exist as long as nobody knows about them. That's just the same "security by obscurity" strategy that has been tried and failed time and time again.

[sudosysgen]

> "defenders have to be right 100% of the time, while attackers only have to be right once"

If you have an adaptive system that can react to attacks flexible (say, your own AI agent), then no, that's not correct. It is correct in the classical conception of cybersecurity where the defender is basically static.

[pastel8739]

Doesn’t this “adaptive system” just become part of the static defense? The same way that a bit of code that checks passwords against a db is “dynamic”, the options are either to beat the dynamic system (guess/phish a password, trick the AI) or find a way around it (use “forgot your password”, find a place that isn’t covered by the endpoint protection feeding the AI). I don’t see how inserting an agent somewhere fundamentally changes anything

[Der_Einzige]

They downvote you cus you're right.

[paxys]

This makes no sense. Why do defenders get to pick the surface area? You can be attacked from anywhere.

[shepherdjerred]

I think the point you're trying to make is that you can always make your exposed surface area smaller

But that, of course, is not going to survive contact with reality

[foo12bar]

And here I thought management picked the surface area.

[xiphias2]

Not really, the only reason I (or any other programmer) haven't ever hacked into a system to make my life easier (not to do bad things) is because it's illegal.

It was always easier than making a system secure.

[artrockalter]

I know the company I consult for (not cybersecurity) is not in these programs and if attacked would need to use open weight models.

[ajyoon]

Did the company apply for access? This is either a problem with your company or the trusted access program. In no way does that suggest the solution is total unfettered access for everyone.

[verdverm]

Everyone needs access to Ai enabled security for defense. A "trusted access program" creates exclusiveness in the hands of Big Ai duopoly. I do not trust them at all

[heyjstn]

are you working at anthropic? you're literally repeat what that freaking Dario say everyday

[paxys]

> Only by using the open source GLM-5.2 were they able to survive an attack

They did not "survive" anything. The attack was long done, and they used GLM after the fact to parse logs. Having a more powerful model would have changed nothing.

If every attacker and every defender has AI with the same capabilities then attackers are going to win 10 times out of 10.

[varenc]

You're ignoring the asymmetry with security. The attacker just needs one exploit chain, whereas the defender needs to block every avenue. Open access to models with no guardrails greatly benefits the attackers more than the defenders.

Imagine what a god-level hacking AI could do. It could find a full 0-click to root exploit chain in iOS. Attacker unleashes a worm that infects a phone, instructs that phone to send the same attack to all of its contacts, and then physically destroy the phone by turning off all thermal throttling. Might even be possible to make it catch fire.

Or find a remote exploit in Tesla cars and make their autopilot go on murdering rampages. (that one is from a movie)

[Majromax]

> The attacker just needs one exploit chain, whereas the defender needs to block every avenue. Open access to models with no guardrails greatly benefits the attackers more than the defenders.

I see it as the opposite, where the attacker needs to find an exploit chain whereas the defender can block any link.

In this model, the balance of convenience favours the defender. The defender presumably has access to the source code and configuration, so their scope of action is much larger than the attacker that must find vulnerabilities in a particular configuration.

I think that the different views might relate to different prior assumptions. If we assume that each layer is mostly secure but may have a small number of latent vulnerabilities, then it should be relatively easy to find and fix those to create a perfectly secure layer. If instead we assume that each layer is mostly insecure but chaining vulnerabilities is time-consuming then the land favours better-resourced attackers.

> Or find a remote exploit in Tesla cars and make their autopilot go on murdering rampages. (that one is from a movie)

In the worst case, air gaps and fixed contracts for information handling cover that. Like any other domain, a car can be remotely exploitable only when untrusted information can influence behaviour inside the secured region. Unfortunately, the convenience of OTA updates and 'cars as tech' rewards velocity at the expense of defensive design.

[vultour]

I have yet to see someone explain why they even needed an LLM to figure out what's going on, other than further proliferating this industry AI psychosis. Are their engineers actually so incompetent that they can't read a bunch of logs without AI? Here I was thinking these fancy AI companies are only hiring the best and the brightest, but apparently 7 rounds of leetcode does a number on your hiring process.

[alienbaby]

? There were not models fighting each other, attacker and defender. I dont quite follow what your getting at.

[akersten]

huggingface asked the frontier models to help them analyze the attack and lock down their systems

the frontier models refused because their cyber detector went off

they had to use GLM 5.2 instead

[paxys]

They used GLM to parse logs after the incident. There was no sci-fi AI vs AI battle.

[usef-]

Yes, to parse logs afterwards and understand, it wasn't active defence from what I've heard? Definitely embarrassing for the closed vendors though (they've since added hugging face as a trusted vendor)

[marcus_holmes]

One of their learnings from the incident was that they should have a local (i.e. not hosted), open, and capable model on standby that can respond to future incidents swiftly.

[adastra22]

The bioweapon thing is absolute movie plot fiction. Go speak to some biologists about this and they'll set you straight.

Cyber capabilities go both ways. Better offensive capabilities means better penetration testing by white hat security experts, which leads to better protections.

[gr_norm]

The effective altruism/rationalism/AI xrisk people have always had a shockingly poor grasp on subjects outside computer science, despite their attempts to speak on them. I don't blame the actual biologists and chemists working at the frontier labs for wanting to skim a few bucks off all the money flying around, though! I know a couple who've had not-so-kind words to say about their employers' intelligence.

I suspect there's at least some "telling the bosses what they want to hear" going on. A massive financial incentive exists to exaggerate and fearmonger even internally to the company, because it makes you and your job seem more important.

[uncivilized]

They also have a shockingly poor grasp of computer science.

[xyzsparetimexyz]

What's the explanation for why a bioweapon couldn't easily be made? As someone who doesn't have access to micro biologists

[adastra22]

You can go download the smallpox genome. You don't need AI to do that. The problem of creating bioweapons remains that it requires very, very meticulous lab work under careful conditions, and a lot of experimental knowledge that the bioweapons facilities probably have but LLMs do not. There isn't an actual mechanism here by which you can mix together a few test tubes and come up with a killer virus.

[xyzsparetimexyz]

You mean I can't use open up smallpox.bp with a hex editor and then 3d print it?

[qnleigh]

So many commenters are asserting this but no one is giving an argument or references. Cults have been able to make sarin. The DNA sequence for smallpox is pubicly available. Where are you getting your confidence that LLM-assisted bioweapons are of no concern?

[gr_norm]

Not only is the DNA sequence for smallpox available, but since 2018 there's been a well-documented end-to-end synthesis procedure for the very closely related horsepox virus [1]! No LLMs needed. Caused quite a stir in the synthetic biology community back then.

The world has not come to an end, of course, because even with peer-reviewed and experience-driven (rather than hallucinated and therefore dangerous) instructions detailing obstacles encountered during synthesis and how to overcome them, actually going out and acquiring the materials and ability to use them sufficiently skillfully is another matter entirely. Biosecurity is an important topic, to be sure, but what the AI labs have to say about it (or anything) at this point does not necessarily survive contact with reality.

[1] https://journals.plos.org/plosone/article?id=10.1371/journal...

[qnleigh]

I'm not talking about what the AI labs are saying and it. I don't know what they have been saying and I don't care. I care whether LLMs a few generations from now will be about to bring these dangerous capabilities to the masses. Right now, given the progress of open and closed-source models, it seems inevitable.

[urams]

Half or more of Hacker News is constantly pushing the idea that frontier LLMs are stochastic parrots and basically useless, even in the face of the Hugging Face incident which most people also would have described as movie plot fiction until it happened. I expect biologists are even less well versed in the abilities of frontier models.

What's more, you can just try a jailbreak on a model yourself to see just how much detailed, step-by-step direction you can get to build bio-terror materials.

[idontbelonghere]

I don't understand the significance of the HF incident. The hacking robot was told to achieve a certain goal and in order to do it, it hacked someone. The ostensible major event here is that it broke out of its sandbox, but how are we supposed to interpret that? AI often misunderstands or doesn't strictly follow the orders you give it, so why is it such a big deal that it didn't follow the rules this time?

[adastra22]

Agree with you here. AFAIK we don't really know what cybersecurity task it was given in that sandbox, but it's not a very large leap to assume part of this task involved hacking. The intention would have been within the sandbox, but what does the AI know? As far as it could tell it just reached Level 2.

[adastra22]

The chalenges are in execution, not availability of information.

[gck1]

I've got zero knowledge of bio, so can't answer that. But with cyber the answer is very simple - the attackers already have more cyber-offense capabilities and there's no putting it back.

Open/closed doesn't matter that much. You can get closed models to do a lot of cyber harm, even with all the guardrails, which currently are heavily skewed towards more false positives.

The only effective control is to level the playing field. If both offense and defense have access to the same capabilities, then we're relatively back where we started.

If you want to ensure chaos, then you do what Dario is proposing to do - create gates that attackers can bypass and defenders can not.

[ajyoon]

In cybersecurity, a level playing field favors the attacker. Trusted access programs give defenders access to tools they need. It's not perfect (because there is an extremely long tail of defenders who are not technically savvy enough to get on these programs and use the tools), but it's better than total access.

The bio angle is very important here too; in that context the imbalance favors the attackers much more.

[gck1]

> In cybersecurity, a level playing field favors the attacker

Yes, but didn't it always? Hence why my position is that this will get us back to relatively where we were pre-LLMs.

And I don't know what Trusted Access programs give to defenders, because as a defender who has credentials, connections, but no deep pockets and no high ranking passport, it only gave me silence. I fail to see how this is better than total access.

I don't think the world where defense is given to those that "deserve" it is the world that we all want to live in. Which brings me back to the starting point - attackers are almost completely unaffected. If I masquarade as an attacker, I get way more capabilities already.

[ajyoon]

> Yes, but didn't it always? Hence why my position is that this will get us back to relatively where we were pre-LLMs.

Trusted access programs are asymmetrical, and so at least for the time being they give critical parts of the stack an advantage. Total access would not be a return to the status quo; attackers can easily make thousands of agents crawl the web for soft targets well before defenses can be shored up. There are millions of targets out there who won't use AI to improve their defenses for years, if ever, due to institutional slowness (like hospitals).

> attackers are almost completely unaffected. If I masquarade as an attacker, I get way more capabilities already.

What do you mean by this? If guardrails are an obstacle to your defense, they are just as much an obstacle to attackers. I completely understand and agree that trusted access programs are not perfect and leave a lot of people and institutions out. This means trusted access programs should be improved, not that we should throw the baby out with the bath water.

[gck1]

It took me a few hours to find some very questionable communities, which in turn gave me access to:

- Ways to obtain cheap guarded-AI tokens that are not linked back to me and with no danger of getting my legitimate accounts banned

- Ways to get rid of guardrails and have models work on things they wouldn't otherwise work on.

The attackers were already in these communities long before I knew they existed, they already had the advantage. Ones with enough reputation probably have access to even more information and tools than I do.

It is true that these communities exist because guardrails were put in place, so yes, it is slowing them down too - as in they can't just put in their CC on claude.com and hack a hospital. But attackers are much better at finding these communities and utilizing resources available there than defenders.

Personally, I don't have any ethical concerns of utilizing these resources when I put them to actual defense, but I know many people that would, leaving them at a disadvantage.

My point is that there's only one guardrail that will effectively contain the threat the models pose, and it's in direct conflict of the big 2's goals - pull the models from worldwide access completely. Strict KYC and all. And it would only last for so long anyway.

[niwtsol]

I feel like so many people miss what you are saying here. The attackers are at such an advantage because of time. At t0, attackers can go and try and find so many attack angles. These traditional companies (defenders) can't just go to a model and say "fix all my things!" and ship it, way more complex in practice.

[CubsFan1060]

I think you are trying to argue that you can limit the open models.

If China is ok with open models being open... they will be. An attacker isn't going to be deterred by a US law saying they can't use them.

I guess my point is that if China is ok with open models, then, the attackers will have them regardless of any laws in other countries. Restricting them, in that case, doesn't seem to accomplish much?

[ajyoon]

You can at least make it harder by requiring US clouds to only serve models with guardrails, and encouraging other countries to do the same. But yes, the underlying issue is the models being open in the first place. I'm sure if the US wanted to, it could come to some agreement with China about this.

[overgard]

> To everyone here pushing for total proliferation of open models -- what should be done about open weight bioweapon and cyber-offense capabilities?

Nothing should be done. These things are trained on public knowledge. The dangerous information is already out there. If someone wants to do something horrible, making it slightly inconvenient isn't going to do much. Hackers and terrorists existed before AI. Just as an example, it's no secret how you would build a nuclear bomb. The practicalities of doing so are much harder, obviously, but the knowledge of how they work and what it would take to make one is not a secret. Security through obscurity has never worked!

[CapsAdmin]

I think nothing can be done anymore, but I don't buy that security through obscurity never worked in practice. A better way to look at this is effort rather than obscurity. The effort it takes to do something with AI is going down, not up.

Almost everything was possible given you put in the effort, but few people possess the will to put in the effort AND pursue a malicious goal.

I think an obvious example is all the fake ai content flooding the internet made to trick people in exchange for money (ad revenue, scams, likes, etc). This existed before ai, but I think it's fair to say pumping out content now requires less effort than it did before.

Most physical locks are an example of security through obscurity/effort. You can after all just pick a lock if you go through the effort to learn the skill. But once a universal lock picker is made available to everyone, you will simply see more locks getting picked.

[rubslopes]

If this is really the risk, then we should approach LLMs like atomic bombs: the US should reach out to other nations so they all agree on no one developing any more AI models. That's the only way you could possibly convince another party to stop. The US should set the example, not conveniently keep all the spoils.

[boinkboink78912]

> If this is really the risk, then we should approach LLMs like atomic bombs

A complete failure at actually preventing non-proliferation.

[paxys]

Uh, that is not how the nuclear race went. The winners kept developing theirs and stopped everyone else by the threat of said weapons. The AI race is going the exact same way, just with China instead of USSR this time.

[rubslopes]

Maybe my phrasing was confusing. I didn't mean that that was what happened, but the ideal approach to stop atomic bombs (at least in my view).

[idontbelonghere]

Experts say Iran is 6 weeks away from making Claude 2

[verdverm]

unfortunately, the US has incentivized the opposite behavior for atomic bombs and we are seeing moves towards greater proliferation

I would not be surprised if the same incentives are created by the US for Ai

[manoDev]

This Pandora box is already open. Any argument about guardrails now are only attempts to create an artificial monopoly or keep this power in the hand of a single nation state, and _that_ is the absolute worst, most authoritarian future possible.

[monk_grilla]

I think you're right. "Guardrails" as a concept has always struck me as a band-aid solution which any sufficiently motivated actor will circumvent by either bypassing them or using unrestricted, open-weight models.

In order to start securing and accepting our new reality we need to assume that capable, open-weight, unrestricted models will be widely available, and that their 3-6 month lag behind frontier proprietary models is just our forewarning of what attackers will soon be capable of. Trying to legislate against or control trade in such a valuable commodity is folly.

I also think that lag is going to shrink over time as the open-weight labs get more capable, acquire more hardware and the plateau starts to emerge.

[paxys]

How is it already open? There has been ONE successful AI-driven cyberattack, and that was done by a model in testing that no one has access to. What would the picture be today if OpenAI and Anthropic had released 5.6 Sol and Mythos to everyone with no cyber restrictions (which is what everyone here was advocating for)?

[boinkboink78912]

> There has been ONE successful AI-driven cyberattack

Not according to Anthropic https://www.anthropic.com/news/disrupting-AI-espionage

[hnfong]

Chinese AI companies are already releasing frontier-ish models every other month. Their rate of progress does not seem to be slowing down. Nobody here can stop them from progressing. Not you, not me, not the USA government.

The "best" thing the US government can do is to build a Great Firewall to wall off the "existential threat from China". I'm not an American so if you guys decide to do it, good luck.

[e_l]

> what should be done about open weight bioweapon and cyber-offense capabilities? Is it simply the cost of freedom that we should allow attackers to access these tools?

In short, yes, it's the price of freedom. As others have said, blocking these models won't stop the "bad guys", but will hinder defenders researching/responding to bioweapons and cyber-offenses.

But you're right that there's a lot of difficult nuance aand we should think carefully about its implications. So here's another nuance to think through.

If AI is as powerful as some believe, then there's much greater danger to give a small subset of society the privilege to gate keep who has access to these tools.

"Power corrupts and absolute power corrupts absolutely." Lord Acton

[tacet]

the bioweapon panic is funny. "oh, yes i know nothing about bicrobiology but i will follow instructions of synthetic text generation machine on temperature 1 about how to design a lab to not kill myself while brewing organisms that will kill myself if i make mistake"

There is nothing that special about bioweapons, there are plenty of bacteria that will kill you just fine. Americans even have free samples on their salad.

[consumer451]

> There is nothing that special about bioweapons.

The reason that madmen and terrorists choose kinetic weapons is because the knowledge and materials are more readily available... of and also that even terrorists are likely aware that their own people would suffer. As the knowledge and tools for playing with CRISPR-style biological legos become more widespread, we come closer to the Great Filter, where one person could kill billions.

Even our normal mad leaders have agreed that bioweapons cannot be allowed:

https://en.wikipedia.org/wiki/Biological_Weapons_Convention

[johncolanduoni]

A halfway decent synthetic biology lab (no need to invoke CRISPR) can make e.g. smallpox without a sample of the original disease, just from the gene sequences. Basically all state actors could do this if they wanted to without an LLM. What barrier that a terrorist organization faces today to having a functioning synthetic biology lab does an LLM actually solve?

[consumer451]

> Basically all state actors could do this if they wanted to without an LLM.

The issue is non-state actors. That moves it from a ~hunderd to many billions.

BTW, according to my FOSS religious beliefs, I should be making the other side of the argument. This whole thing is tough.

[mnicky]

With current gaps in DNA synthesis screening yes. But this will be improved in the future hopefully.

[tacet]

i meant it in the sense of arcane knowledge model could have that would make it simple for anyone to brew up in cheap lab while managing to not infect themselves over and over.

"at home" bioweapon panic has been around since crispr and rna synthesis got available to amateurs.

[consumer451]

I appreciate the reply. I did not mean to be dismissive at all. In the interest of a good exchange, I have to say:

I really want open weight models. Otherwise, I see no other path outside of the labs eventually not being allowed to/wanting to release model access at all, and instead just eating all the verticals. That would be a horrible near-term business outcome.

[rstuart4133]

> what should be done about open weight bioweapon and cyber-offense capabilities?

Like the others here I know almost nothing about bio weapons, but I think perhaps the fact that smallpox's genome sequence has publicly available in scientific databases like GenBank for 30 years is relevant. That horse bolted a long time ago.

[impossiblefork]

There's no stopping bioweapons. Bioweapons are easy. The reason bioweapons aren't built is because very few biology nerds with sufficient lab skills are evil; and just having an LLM won't give you the lab skills to do it.

Anyone who can publish a gene technology/biomedicine paper can make a bioweapon. If you wrote a paper about how to make a bioweapon easily, it would be unpublishable not because of any danger, but because there wasn't enough novelty.

[__MatrixMan__]

The scariest outcome here is that a bunch of lunatics get ahold of a capable model and use to to harm the rest of us, who are at a disadvantage due to just how capable the model is.

But that's what's happening. The people in charge are a bunch of lunatics. However nice it would be to prevent them from having harmful capabilities, that ship has sailed. The best we can hope for now is preventing them from having supremacy, and that's what open weight models do.

[boinkboink78912]

> Is it simply the cost of freedom that we should allow attackers to access these tools?

Yes, it is inevitable that open weights models will happen. Through legitimate means or leaks, the stakes are simply too high once these models get powerful enough. Furthermore, state-sponsored attackers will always have access to these capabilities. The best we can do is give a lot of preparation to the defenders.

> Biasing toward supporting openness makes sense and is a good instinct, but it's incredibly naive to be absolutely in favor of it in every circumstance without seriously thinking about its implications.

I find it funny that Anthropic's entire argument for building RSI is that it is inevitable, and therefore we should commit to building it first and doing it safely, and yet they don't apply their own logic to open weights models.

[qnleigh]

I'm dismayed that I had to scroll past so many cynical cheap shots to find a comment that actually addresses the core point. I have yet to hear a single compelling plan for how we will prevent bioweapon development or massive hacking campaigns. For those who are skeptical of Dario's motives here, it's not enough to call out apparent hypocrisy, you need to suggest an alternative plan that addresses these concerns.

[nevertoolate]

> I had to scroll past so many cynical cheap shots

I haven't read cynical comments, just ones pointing out that the article is cynical itself.

> addresses the core point

No it doesn't address anything, it is fear mongering question.

> I have yet to hear a single compelling plan for how we will prevent bioweapon development or massive hacking campaigns

Me neither, I just see marketing campaigns trying to raise valuation of a pre-IPO company.

> you need to suggest an alternative plan that addresses these concerns

I suggest that Dario stops writing marketing letters and start organizing a mostly neutral expert organization to propose solutions.

Also notice that as EU citizen I don't trust a US pre-IPO company's CEO with conflict of interest to suggest solution on resolving global security matters. Especially since he admittedly has no control over how the technology of his own company is deployed in global conflicts[1]

[1] https://www.forbes.com/sites/antoniopequenoiv/2026/06/10/ant...

[txrx0000]

I do seriously want general intelligence to be widely available with no guardrails, and there are very good reasons for this. If you want to read about it:

https://news.ycombinator.com/item?id=49078376

----

And related thoughts on past posts:

https://news.ycombinator.com/item?id=49034988

https://news.ycombinator.com/item?id=48516722

[CJefferson]

The problem is you can't ban open models.

All you can do is say that Americans have to pay whatever stupid prices OpenAI / anthropic / Google / Grok wants to charge you, while China uses, and attacks with, open models.

[le-mark]

> Is it simply the cost of freedom that we should allow attackers to access these tools?

Bad actors WILL have access. The question is will these mega corps stop innovation?

[valcron1000]

> what should be done about open weight bioweapon and cyber-offense capabilities? Is it simply the cost of freedom that we should allow attackers to access these tools?

Yes, in the same way that we have E2E encryption which allows bad actors to distribute content beyond human horrors.

[rnewme]

What's the magic dataset LLM had that bad guys can't dig up online? Do LLMs train on deep web content? Or leaked lab data?

[fidotron]

> what should be done about open weight bioweapon

Does not exist. What has in fact happened is some cults had bioweapons programs but any failure points were at deployment. (Aum Shinrikyo https://en.wikipedia.org/wiki/Tokyo_subway_sarin_attack and https://en.wikipedia.org/wiki/1984_Rajneeshee_bioterror_atta... )

> and cyber-offense capabilities?

You mean defense. That's how things get hardened. Anyone that was working during the XP era before Service Pack 2 knows what that was like, but it's very manageable.

The bigger real problem here is hardening like that would remove the opportunity for intelligence agencies to spy on everyone.

[jefftk]

I don't think the Aum case points the way you're describing: they used a non-pathogenic strain of anthrax because they didn't know any better. That's a knowledge failure.

But even then, the debate isn't about whether open weight bioweapons exist today: it's about whether they will exist in the future. I think Amodei's argument here makes a lot of sense: "what I believe currently keeps us safe in biology is not 'defenders', or even the availability of materials, but a negative correlation between intellectual capability and desire to commit catastrophic harm. Previous technologies like internet search or even DNA synthesis were nowhere near powerful enough to break this correlation, but I worry that at its current rate of progress, AI will do so very soon."

(I'm not just spouting off; I put my time where my mouth is. I used to work in big tech, but I left for a much less well-paying job building an early-warning system for engineered pandemics.)

[fidotron]

> I don't think the Aum case points the way you're describing: they used a non-pathogenic strain of anthrax because they didn't know any better. That's a knowledge failure.

No, check https://en.wikipedia.org/wiki/Matsumoto_sarin_attack

There are a lot of interviews with former cult members around. They had armed helicopters, a testing station in western Australia, produced piles of sarin. This wasn't a lack of science knowledge that screwed them up, they notoriously involved the elite class of Japan - it was a whole other category.

There is no link between AI and bioweapons that makes this stuff any more reasonable than availability of detailed descriptions of nuclear reactors enables us to be purifying weapons grade plutonium in our yards.

[jefftk]

> > they used a non-pathogenic strain of anthrax

> No, check https://en.wikipedia.org/wiki/Matsumoto_sarin_attack

That's a different attack. I'm talking about their 1993 anthrax attack: https://pmc.ncbi.nlm.nih.gov/articles/PMC3322761/

Analysis of the 48 suspect colonies confirmed them to be B. anthracis ... This genotype was identical to that of the Sterne 34F2 strain, used commercially in Japan to vaccinate animals against anthrax.

They used a vaccine strain because they didn't know any better. Even members of the elite can make mistakes, especially when operating outside areas they know well!

(This was not the only thing that went wrong, but several others were also knowledge failures.)

[fidotron]

You and the other are both missing the point. That's not a knowledge failure, it's a failure in how your operation is strategically executing. They were essentially practicing, and what did they learn? Change to sarin and even VX, for which they didn't need AI.

AI isn't going to help you get from nonpathogenic anthrax to pathogenic anthrax either. All it might do is tell you to try sarin or VX earlier, but these present different problems.

The idea that there are people in the world wanting to execute bioweapon attacks that are somehow gated by a lack of access to AI is utter hysterical nonsense that should be clearly pointed out as such.

[rescbr]

Still, somebody heavily funded this thing for too long, and I very much doubt that we don't have the surveillance apparatus in place today for these things to get unchecked.

Stuff is known but not acted upon for various reasons.

[ajyoon]

You admit that some attackers have the inclination to use bioweapons. Why would they not use the best tools at their disposal going forward?

From the WSJ the other day:

> After OpenAI enhanced the brain power of its chatbot last summer, hundreds of users worldwide began asking it how to make and deploy biological weapons and poisons.

https://www.wsj.com/tech/ai/openai-chatbot-biological-weapon...

On cyber, the attacker/defender asymmetry strongly favors attackers. There are millions of soft targets on the internet which do not have the savvy to use AI to shore up their defenses.

[fidotron]

> Why would they not use the best tools at their disposal going forward?

Because AI doesn't solve any of the problems any attacker would actually have. It's a classic case of nerds not seeing the actual problems because they involve reality.

It's worth pointing out that those bioweapon attacks I linked to also predate widespread access to the Internet, and there was similar scare nonsense about that.

> On cyber, the attacker/defender asymmetry strongly favors attackers. There are millions of soft targets on the internet which do not have the savvy to use AI to shore up their defenses.

Do you think they are not being exploited today? The reason they aren't more exploited is there really isn't much to gain from doing so.

[ajyoon]

jefftk addresses your bio thought well.

> The reason they aren't more exploited is there really isn't much to gain from doing so.

This is incorrect. The long tail of soft targets aren't being exploited more because attackers are bottlenecked on labor. AI removes exactly this bottleneck.

[fidotron]

> This is incorrect. The long tail of soft targets aren't being exploited more because attackers are bottlenecked on labor. AI removes exactly this bottleneck.

No, it's because the targets are worthless.

You aren't going to be able to mine Monero or run LLM botnets on forgotten cameras in basements. There is nothing to be gained from such targets, soft as they are.

Besides the new defensive AI entertainment makes dealing with wherever those things phone home far easier. Possibly too easy for plebs to be allowed access to.

[wilde]

HuggingFace was only able to defend themselves with open models. No need to project into the future. Look at the timeline of events for that incident.

[dools]

The difference with open weight is that everyone has access to the same weaponry

[verdverm]

> what should be done about open weight bioweapon

The same thing we do about bomb making today, certain ingredients are restricted and/or monitored. Bioengineering is a bigger lift to operationalize.

In other words, don't ban knowledge, make certain applications or ingredients illegal or highly regulated.

[jackdeansmith]

And what are those ingredients for biology, which can be constrained as effectively as uranium enrichment? I'd argue there isn't anything which can easily be restricted or monitored.

[verdverm]

I was referring to non nuclear bombs like c4, artillery shells, and missile payloads

[dolebirchwood]

> what should be done about open weight bioweapon and cyber-offense capabilities?

If the model is capable of it, then it was in the model's training data, which means it was on the internet or published in books made available for consumption. So if any member of the public could have gotten their hands on that information, so be it. If the knowledge was too dangerous for public access, then it should have been highly classified and never found its way into the training data. Tough shit, frankly.

[paxys]

Was the proof of the Cycle Double Cover Conjecture in the training data?

[adastra22]

The threats are BS, but fyi models have repeatedly shown capability to produce novel things that are NOT in their training set.

[dolebirchwood]

Sure, based on predicate knowledge.

[baddash]

maybe instead of worrying that people on the internet will be good at coding, we could start writing memory safe apis. almost all cves are fixed by using rust

[gorgoiler]

If everyone has access to the same offensive tools, everyone is able to run their own pentests and patch themselves before the bad guys get to them.

It’s like a vaccine where you get to try a medication based on the original pathogen by performing a dry-run on a backup of yourself already in a hospital ward.

Open models aren’t like firearms. If everyone has a gun the mall parking lot is a much more dangerous place because the consequences of using a firearm are so dire, even if you’re in the right.

[pbasista]

> what should be done about open weight bioweapon and cyber-offense capabilities?

In my opinion, the governments should deploy open-weight AI countermeasures. Because it seems to me that it is impossible to efficiently fight AI-powered criminals without AI.

When only AI-restricting regulations would be put in place, the criminals would, in my opinion, just ignore it. We as a society have a difficult time tracking even the illegal gun or drug dealers. I cannot imagine how could one hope to "regulate" something that can be downloaded as a file and run on a computer.

These AI countermeasures should be open because it provides transparency as to whether the countermeasures actually work. Independent testing, tuning, refining or retraining is then possible.

If the closed models were used instead, their provider could at any point in time shut down the entire operation. Or sabotage it under the hood.

The important part is that with the closed, black box, proprietary models, one can never know what they are being served.

[BeetleB]

Everything you said could apply to computers many decades ago. Think of the nuclear fission simulations our enemies could carry out!

We'll be fine.

[Madmallard]

Too bad. We'll have to deal with it. There is no way to stop the weaponization of models now.

But more people having access to the potential tools for defensive is the best possible scenario.

Every other scenario is worse off for everyone except for those with enough money to do something about it.

[waterTanuki]

The moat never was and will never be the models, it's the hardware. This is exactly like nuclear weapons: The recipe for a nuke isn't a hidden secret. Getting the infrastructure and materials is completely unreachable for non-state and non-corporate actors. This idea of a "rogue individual" using a frontier model to develop a bioweapon is a complete myth, because anyone with the capability to run the models without guardrails has to answer to/be audited by some entity already.

[fwn]

There is also a whole second category of immense risks of having US companies gatekeeping offensive capabilities, especially for us here in Europe. The centralization/privacy/kill-switch concerns that come with it are a huge AI safety dimension.

I'd rather have a level playing field within a phase of adaptation and hardening regarding cybersecurity issues than a constant dependency on the US, maybe grabbing Greenland today, maybe "extracting" our president tomorrow.

The delta between privileged capabilities and open weight capabilities alone already is a massive, unaddressed AI safety risk.

[pylua]

The software industry should be ashamed by the number of exploits that ai can find in software. It’s really an embarrassment.

The software has to be built better.

[doginasuit]

I'm curious if you are a coder and have used an LLM to review your code. It is like something like shining a black light around a hotel room, and that seems to be the case even for highly regarded software.

It is really easy to have tunnel vision while coding. LLMs have a working memory with a capacity an order of magnitude greater than ours. I wouldn't trust an LLM to write the code, but at this point it is malpractice not to use one for review.

[pylua]

I have been, yes. For a field that has engineering in the name there sure has been a lot of critical mistakes.

You have to call a spade a spade — the profession accepts this sort of tradeoff in the name of speed and cost.

A well designed system would have never allowed those mistakes to occur. I feel like using an llm to catch these sorts of things is just because it wasn’t built right in the first place.

I think ai systems will be able to build systems of abstraction that are formally verified, and we won’t be needed(eventually).

Right now it’s being used as a bandaid.

[pastel8739]

Formally verified against what spec?

[naiveter]

The one liner leadership keeps asking for.

[marcus_holmes]

Every single software engineer in the industry agrees with you.

Every single project manager disagrees.

Don't blame the engineers, we were specifically instructed and paid to build things fast and cheap, and every time we argued for good we were shouted down.

[pylua]

True. It has been a race to the bottom for lowest cost as long as the quality meets the bare minimum. LLMs can go through and find all the nails sticking out pretty easily.

[zarzavat]

There's a difference between:

> Something should be done

and

> Something can be done

In this case, nothing can be done to stop bad actors from using open models. As the article points out, the US can only feasibly prevent US businesses from using open models.

The US can attempt to stop those models from being trained in the first place but good luck with that.

[vitalyan8184]

>To everyone here pushing for total proliferation of ...

...general-purpose computers

...unbreakable encryption

...unbackdoored communication

...unkillswitched vehicles

...unsurveiled dwellings

>what should be done about ...?

nothing

>Do you seriously want this level of capabilities to be generally available with no guardrails?

yes

[jackdeansmith]

What about uranium enrichment?

[jjfoooo4]

Requires some pretty tough to get raw materials and equipment, to say the least

[davrosthedalek]

Because of regulation, not because they are intrinsically hard to get.