[fidotron]

> For all I know (I don't know a lot about biology), it's possible to develop a virus that spreads as easily as measles, but has the lethality and long induction period of HIV. If that's true, and if LLMs would make it easy to develop such a virus, I see how no-guardrails LLMs could be truly disastrous.

This is the specific example in the latest doomerist wave I have so much trouble with: by what mechanism is an LLM going to help you design and manufacture a novel bioweapon? We know they cannot do spatial reasoning, and if they were this good at bio stuff then drug research would be absolutely out of control.

Similarly, how are you supposed to magically get access to everything required to actually manufacture the offensive organism in question? Are we all doing CRISPR in our kitchens now?

If you are that concerned about this threat then . . . monitor biolabs, not AI. It is so detached from the real state of things as to be bizarre.

[alach11]

I think some AI safety advocates have been arguing for great controls around nucleotide synthesis (e.g., impose greater data collection requirements or advanced screening for harmful sequences). It's surprisingly easy to order synthesized genes delivered to your doorstop. All you need is a credit card and a nucleotide sequence file!

Developing a bioweapon is already within reach for experts with sufficient funds, time, and motivation. That's more or less what gain-of-function research already does! Fortunately the barrier to entry is high enough that you don't see like... Boko Haram doing this. Or a lone-wolf nutjob. LLMs could definitely lower that barrier to entry though.

[fidotron]

> I think some AI safety advocates have been arguing for great controls around nucleotide synthesis (e.g., impose greater data collection requirements or advanced screening for harmful sequences). It's surprisingly easy to order synthesized genes delivered to your doorstop. All you need is a credit card and a nucleotide sequence file!

So why have the discussion about regulating AI at all if that is the actual problem? Surely instead of blanket blocking everyone's AI models it makes more sense to control the potentially problematic sequencing?

[pu_pe]

Because there are many legitimate uses for nucleotide synthesis, and we would have to slow scientists down by adding more regulation. The other problem is that it's not like it's very difficult to do it, so a motivated person could still do it even without access to these services.

[intorio]

> by what mechanism is an LLM going to help you design and manufacture a novel bioweapon? We know they cannot do spatial reasoning, and if they were this good at bio stuff then drug research would be absolutely out of control.

I guess my question would be how good the LLM is at writing a program that is good at spatial reasoning? If it can create a valid test suite from existing art on the subject it might be able to bumble its way to something specialized that it can then direct at a higher level.

Isn't it that kind of 'rapid self-improvement' that is the concern expressed in the linked statement?

[fidotron]

Have you tried to make LLMs do that? Unless it's implementing previously developed algorithms they screw it up so repeatedly it's incredible. (And even when it is clearly defined it's obviously their weak spot).

The "rapid self improvement" is going to be ever further down the LLM rabbit hole where brute force, tenacity, and symbol juggling beat having a reasonable model of anything else. This is enormously more likely to recommend improvements to its own architecture than to develop a sudden grasp of molecular biology.

There isn't an AI bubble, but I'm increasingly convinced there is an LLM bubble, and that this noise is to do with preventing people noticing that before they've managed to pull something else out of the oven.

[BugsJustFindMe]

> We know they cannot do spatial reasoning

Everything else aside, and regardless of whether that claim is currently true, I would ask you to be aware that that statement requires a way of thinking that ignores predictable future developments. Whatever the capability of any systems currently released, there will be more capable ones tomorrow, and the most you can say is that we have not mastered training spatial reasoning yet, not that we can't or won't.

> Are we all doing CRISPR in our kitchens now?

https://www.the-odin.com/crispr-kit/

[Smaug123]

> cannot do spatial reasoning

Opus 4.8 was the first model that could plausibly roleplay a BJJ roll without making a physically implausible move within five conversational turns. Fable 5 was the first model that beat me at the task, where I was the first party to write down an implausible move.

[OkayPhysicist]

You can do CRISPR in your kitchen. It's been possible practically as long as CRISPR has been a thing. It's not particularly difficult, to the point that my biggest counterpoint to the whole AI doomerism argument on the subject is that nobodies done it yet.

You don't need some genius AI to design a bioweapon, you need to do a bit of light reading on bioengineering. The fact that some mad scientist hasn't ended civilization is evidence of the fact that it's not some existential threat that justifies wild levels of government control.

Unfortunately, AI development is dominated by an even mix of 1) Machine God Cultists, 2) Morally Bankrupt Mercenaries. All three agree on the fact that "thou shalt have no gods before me", because it'll make them richer than sin, and allow them to shape the AI landscape in their image.

[sm-silversight]

Can you point me at this light reading? I don't wanna do CRISPR in my kitchen, but itd be cool to know more about it. I thought I needed to be basically pre-med to grasp it.

[SturgeonsLaw]

Here's a step by step guide on how to gene edit rice

https://bio-protocol.org/en/bpdetail?id=1225&type=0

Mostly it's just heating and cooling tiny volumes of liquids