[skinfaxi]

Wait wtf. The mitochondria thing is true.

> Why this chat was flagged This model has safety measures that flag specific phrases. This can happen to safe, normal chats.

> Your message itself appears to be what’s triggering the safety check. Editing it and retrying may help.

[cge]

There’s confusion about the classifiers on Fable. They don’t ban chemistry and biology topics they flag as a potential risk, they ban anything related to chemistry or biology at all. This is intentional, and directly stated on the model card, but seems so absurd that there can be assumption it must be a misreading.

Being a researcher somewhat connected to chemistry and biology, Fable has been the most useless model I have ever tried. Essentially all work has instantly downgraded to Opus.

[stavros]

Anything biology-related does this. It even did it when I asked it how eye color works, or something about frogs.

[estearum]

Probably because the cost of blocking "is mitochondria the powerhouse of the cell" is nearly zero, while the cost of allowing "how do I synthesize the Spanish flu" is approximately infinite.

[areoform]

    >  while the cost of allowing "how do I synthesize the Spanish flu" is approximately infinite

I've heard this sentiment repeated elsewhere, but why? What makes you think that's the case?

Under this rationale, every serious HS textbook has "approximately infinite" risk. That's clearly not so. Why is this any special?

[estearum]

Does every serious HS comp sci textbook give aspiring software engineers the same power that e.g. Claude Code does?

[areoform]

I think the source of this misapprehension is that,

You are comparing wet work in a lab to writing code on a computer.

When you screw up an exploit, you fail to execute the exploit. Famously, just like software's near zero marginal cost of distribution, the marginal cost of failure is nearly zero.

You can screw up an infinite number of times on your way to a successful exploit.

If you screw up with lethal agents in a lab? You die.

Here's a non-exhaustive list,

    Dora Lush died after accidentally pricking her finger with a needle containing lethal scrub typhus while attempting to develop a vaccine for the disease

   A 23-year-old laboratory assistant at the London School of Hygiene and Tropical Medicine, was infected with smallpox after observing the harvesting of live smallpox virus from eggs without isolation cabinets at that time. The assistant was hospitalised and before being isolated, she infected two visitors to a patient in an adjacent bed, both of whom died. They in turn infected a nurse, who survived

   Ebola laboratory infection by the accidental stick of contaminated needle in the United Kingdom

   Researcher Nikolai Ustinov was lethally infected with the Marburg virus after accidentally pricking himself with a syringe used for inoculation of guinea pigs. The accident occurred at the Scientific-Production Association "Vektor" (today the State Research Center of Virology and Biotechnology "Vektor") in Koltsovo, USSR (today Russia).

"lethally infected with the Marburg virus after accidentally pricking himself"

Anything lethal enough to kill other humans is lethal enough to kill you.

And if you don't know what you're doing — and for this argument you're saying this person has to ask a LLM "how do I spanish flu?" then they definitely don't know what they're doing, the number of ways you will die far outnumber the ways you can succeed.

And this, of course, doesn't even cover the cost of equipment, the precursors, sourcing the highly specific materials needed, then setting the equipment up... etc.

The same is true for the Bosch-Haber / Haber-Bosch process, which famously made WW1 possible. Every HS'er learns about the process and the steps. Steps that were classified once upon a time and were the subject of negotiation at the Versailles.

Does that mean a HS'er (or any adult) can set up an experiment that works at 177 times the pressure of the Earth's atmosphere to do anything at any scale without significant infrastructure and help?

The people who can do this are domain experts, and they've been able to do this with COTS stuff since the 1990s, at the very least, for a price of around $2M – https://en.wikipedia.org/wiki/Project_Bacchus . And those people don't need a LLM to tell them what to do. In fact, they're the exact people who'll have access to unrestricted versions of these LLMs.

And from a security perspective, I would bet good money that flooding the FBI's tip line with junk about every teenager trying to learn "what be a mitochondria" does more harm to the effort of finding people who could be planning such a thing than it helps. It takes more resources to go through the mass of false negatives that have now been created as matter of policy.

These experiments have been run. The fictional scenario of someone learning how bioweapons work and conjuring up a plague isn't real and it hurts humanity as a whole to impede the sciences over it.

Because what someone can flail around in / do is learn about immunology / try to "cure cancer" with a LLM and hopefully get started on a long career in medicine. Or, a discovery that matters.

Because in those cases, if and when they do end up at a lab, screwing up doesn't mean death. Just tons of wasted time (and money). And they will fail / screw up. Just look at literally every undergrad in any lab and the expensive messes they create.

-

And last, but not least, yes. Teenage hackers have been a meme for decades.

[frotaur]

First of all, sure if you screw up designing a bioweapon you die, but unfortunately that does not necessarily mean the bioweapon dies with you, quite the contrary it can kickstart its propagation.

Second, in your argumentation you assume that the experiments are extremely difficult or costly. Thankfully, so far, it seems to be the case that they are too difficult (either due to domain knowledge, or to difficulty of obtaining the necessary components/equipment).

But there is no clear reason to think it will remain this way (e.g. crispr allows for genetic engineering which is very cheap). And it appears that, for domain knowledge, capable LLMs are rapidly reducing that barrier. We are not there yet of course, but I think it is crazy to dismiss these concerns, which are very real and crucially are NOT just brought up by the big labs.

[estearum]

No, the source of the misapprehension is that you are ignoring how easy it is to source the input components and combine them into a bioweapon these days.

Correct, there is a small number of people who have been able to do it at high cost for a long time.

Now, there is an ever-growing number of people who are able to do it at an ever-falling cost.

That's the entire issue. Do you dispute that this is what's happening?

> I would bet good money that flooding the FBI's tip line with junk about every teenager trying to learn "what be a mitochondria" does more harm

Did someone propose doing that? Or is this a strawman?