[alevskaya]

I was a genetic engineer for ~20 years and have worked on frontier LLMs for the last 8. I used to engineer viral vectors and studied how to evade human immune systems for gene therapies...

The biorisk scenarios that the AI safety folks flog are fever-dreamed fantasies that have only the most tenuous connection to biological reality. As someone who cares about the real bio-risks of natural pathogens, I get pretty tired of fear-based marketing pretending that AI is a bigger threat than, say, animal agriculture.

[Diogenesian]

It has always been ridiculous to suppose that some organization spent $500m on a microbiology / genetics lab, but ran out of money for scientists, so they have to ask Claude what to do. Or OTOH to suppose a guy in his garage set up a weapons-grade CRISPR lab without anybody noticing.

[alt227]

> The biorisk scenarios that the AI safety folks flog are fever-dreamed fantasies that have only the most tenuous connection to biological reality

Do you not think that at the rate ai intelligence and ability is increasing, this could realistically change one day soon?

[rdedev]

Even if AI gets super smart, it will run into the limits of what we know about biology. Someone will have to do lab experiments to provide more knowledge to the AI. This is different from say building a computer virus or hacking since the AI can do all of these on it's own

[mnicky]

> The biorisk scenarios that the AI safety folks flog are fever-dreamed fantasies that have only the most tenuous connection to biological reality.

As an expert, could you also provide your arguments please?

[inciampati]

Anyone can write out the code for a bad virus. You can go download it from an open repository. It's only through deep interface with the world that the idea for the bad virus turns into an actual bad virus. The fever dreamers will say the LLMs will help you interface with reality to do the bad thing™ which their model let's you do. But you still need thousands to millions of times the effort And once made how do you deliver it in a way that might further your (bad) objectives? Presumably it just makes humans sick. There aren't "targeted" bioweapons, and among humans we are too damn similar for there ever to be. And all of this said, there is almost nothing special about the LLMs' abilities in biology. They only know what we know. They're not being trained autonomously with RL and a robotic wetlab. When that's a thing I'll start to take claims of biology risk more seriously. Right now they have the same logic as the paperclip theory of superintelligence risk. And cynically, it would seem that Anthropic purchased a biotech company and almost immediately decided to lock down biology work with their models.

[Ey7NFZ3P0nzAe]

> They're not being trained autonomously with RL and a robotic wetlab. When that's a thing I'll start to take claims of biology risk more seriously

So if IIUC your point is "they're not good enough at biology right now because they're not trained on it so they're not a threat".

To which I want to answer: "they're not a threat now but I see *no* reason for models not to be trained on biology pretty darn soon unless people like you convince the world otherwise."

Thoughts?

[rdedev]

They are already being trained in biology right now? What he is saying is that they are being trained on what we know about biology currently. AI is going to hit that limit. And there is no way for the AI to gain more knowledge without actually doing lab experiments

[Ey7NFZ3P0nzAe]

> AI is going to hit that limit. And there is no way for the AI to gain more knowledge without actually doing lab experiments

That's a big claim. I believe on the other hand that biology is applied chemistry which is applied quantum physics. And frontier models are so good at biology + biochemistry + chemistry + quantum physics that it's bound to trickle down into biology by sheer knowledge.

And secondly, even without pipettes i'm pretty sure computer simulation can teach us a tremendous amount still.

[rdedev]

Maybe ASI can do the things you claim. Like figuring out why we have an appendix just by looking at our DNA.

Till then I don't think current crop of LLM will ever touch that capability. I'll believe it when I see it

[inciampati]

My point is more about conflict of interest in Anthropic, and certain techpeople types thinking that biology is a useful scapegoat because biologists don't use or understand this tech (they claimed 0.03% of users would be affected by biotech security stuff). So, arguing the world will fall apart because someone can ask your superduper powerful tool for a bad bad virus sequence, or how to do some technique in the lab, and get an answer that's plausible (oh but you never bothered to actually test if it's legit because you don't have the capability to do so).

As for the future... today the LLMs are "trained on biology", in that they read the textbooks, the research, the web.

They aren't trained on biology in the sense of being embodied, autonomously or semi-autonomously driving actual biological experiments. If you come from software, the timescale of these experiments is outlandish. Yes, I am partly saying the LLMs are not good enough today because they need to be embodied and trained for literal decades of lab time before there is even the _remote_ possibility that they could present a novel risk profile that is even a shadow of what the current fearmongering suggests the current models can enable.

And they aren't trained on biology in the sense that they've read the literature, but even 100T token training run only begins to touch the data scales that rather mundane bioinformatics operate at. True multimodal models that work on DNA and human language at high quality haven't yet emerged. We're talking new architectures which are going to arise after the next AI winter.

All of this ignores an even more fundamental point. Cost. If someone wants to make a bionuke, they don't need to use AI. They can set up the right evolutionary context and run quadrillions of parallel explorations of the design space. Directed evolution like this is cheap, well-understood, and insanely powerful. If you actually care about biosafety, we should be doing hard work to surveil gain of function research. Different flavors of LLM use are not going to be a differentiator for the foreseeable future.

[qnleigh]

If you are optimistic about this given your expertise, I am very glad to hear it. As someone scientific but with little background in biology, I've been concerned about the bioweapons angle for a long time (and not because AI companies tell me to worry about it). Can you please explain a bit more about why you are not concerned? Between standard bioweapons like sarin and gain-of-function research on viruses, it's not obviously implausible to me that LLMs a few generations from now won't be able to guide a determined layperson through the steps needed to make something very destructive.

[inciampati]

I think I'm less bothered by the risk because I've actually used these systems to do biological research, to work in bioinformatics and to work in the lab. And while I think that the leverage you get in bioinformatics is very significant, the laboratory work has never felt the same.

At best, you get much, much better ability to understand existing literature. the model itself has a very poor understanding of the physical world and that's masked by its knowledge of things people write about the physical world but it intrinsically doesn't have the same kinds of intuition and perspective that are really required to drive integration and completion in this space.

Is AI an important new tool in biology? Well, yes. Does it cause so much uplift in capacity that some rogue actor without biological research background could somehow destroy the world with a super-bio-nuke? I don't think so. I think that's just as logical a conclusion as the idea that next year one of the new frontier models will be told to make as many paperclips as possible and accidentally boil lake Michigan in pursuit of its goal.

[Jyaif]

> fever-dreamed fantasies

Reminder that what local LLMs are achieving today is the "fever-dreamed fantasies" of 5 years ago.

People really need to internalize that we will eventually have the technology for giving everybody the equivalent of a world class scientist locked in their basement that is willing to do anything.

No one knows the timeline, but it's inevitable (barring societal collapse or some kind of legislation)