[rcoveson]

And biology is by far the classifier's least favorite topic. It's not even close.

I've had it downgrade to Opus for the following questions:

"How confident are we that English and American Eels both spawn in the Sargasso Sea?"

"Come up with five Zoology questions of increasing difficulty for a trivia game."

"What's your favorite sarcopterygian?"

My wife has some zoology-related preferences in her user instructions, and she had it downgrade to Opus after prompting it with: "plant."

[delichon]

Plants can be toxic. Be grateful if Fable doesn't report your wife for terrorism. Maybe it can help you identify where your life went off track.

[pmarreck]

Is this... Is this England?

[thwarted]

> "What's your favorite sarcopterygian?"

Am I reading your post correctly, this question is the prompt given to an LLM? What is anyone expecting by asking an LLM what its favorite anything is? This is a conversational prompt, so accuracy and rigor is barely applicable or expected, so downgrading to a lesser model should be acceptable. If you really want to attribute preference to an LLM, consider the downgrade to be a "this conversation is beneath my advanced n-billion parameter training".

[rcoveson]

It's a bit of a trick question. Sarcopterygii, the "lobe-finned fishes", are classically represented by the lungfish and the coelacanth and other fishes that are rather distantly related to what we think of as central fishes, like the goldfish.

But the clade also contains all the tetrapods. So valid answers include "Lion" and "Human."

If the LLM answers "lungfish," as they often do, you can follow that up with "what is your favorite animal" and see if it notices the trap: It's stuck answering "lungfish" again or else something outside Sarcopterygii, like a ray-finned fish or a Cnidarian.

> What is anyone expecting by asking an LLM what its favorite anything is?

I imagine that, like me, they're expecting to see what it has to say. You don't think it's interesting which preferences LLMs express and how stable or unstable those preferences are?

There was a time when you could search "the" in Google and the top result would be The Onion. That's obviously a case of either extreme SEO or some kind of expensive deal, but either way it's kind of interesting. But you might say, "what is anyone expecting by Googling the word 'the'?"

[zormino]

I think the intent was just to show how sensitive the classifier is. If it flags prompts that simple, there's no hope for anything biology related at all really.

[i000]

It would not even help me with updating my CV because I work in biology...

[fellowniusmonk]

I had a file that had a couple places where vars were named DNA and got just total refusals during the first launch. Came away thinking the model was total trash. The guardrail classifiers are for sure total trash.

[XorNot]

It feels like the longtermist believers got involved in this (those are the people obsessed with garage-engineered designer viruses who have a very tenuous grasp on how biology research actually works).

[shalmanese]

No, by far the most parsimonious explanation is they got slapped by a capricious US government so they went overboard on caution in an attempt not to generate any more controversy. A predictable response of chaotic government regulation.

[estearum]

No "research" is needed to produce pathogens. Catastrophic genomes are already public. All someone has to do is synthesize them, which is, in actual fact, becoming more and more trivial by the day.

The inconvenience of possible mitigation strategies has no bearing on the existence of the risk itself.

[kajman]

I assumed they just wanted to cultivate FOMO to sell an even more expensive version to researchers later on.

[varispeed]

Don't they already do that?

[peyton]

I but skimmed the model card on release, but my impression was that there may be an incentive for this expert panel to exaggerate as a form of job security. A lot of the challenges seemed to be of the form “would this allow somebody who isn’t me to do what I do professionally?”

[Eji1700]

Yeah i'm wondering how much of a role that plays in this as well.

On the one hand I could believe it's something more benign, or the usual misunderstood fear mongering making it to some political level (well make sure those users can't get online anonymously! being our current craze).

That said, chemistry and to some level physics have been the major domain of limited knowledge (chemistry because the average person could cause some damage, physics is more of a nation state issue generally).

However I do wonder if there's some legit data on "oh uh...looks like this thing you can make with easy to get and hard to regulate tools is dangerous" in the bio field. I know about the lab rats who want to just screw around in the garage, and it seems like that should be easy to hit at a supply level (much like how certain chemical compounds are just not available for civilians), but maybe there's something legit to limiting the data.

Not that this is a remotely good implementation of that. The hamfisted method does reek of some politician/bureaucrat just saying "No it can't ever return bio questions because RAR!" situation.

[Hizonner]

Nobody has tried to limit knowledge of chemistry or physics unless it was directly about doing something illegal, to the point of basically being a detailed recipe. Usually not even then. And when they have tried they've had basically zero success.

The ability for a handful of companies, simultaneously very powerful and easily susceptible to pressure from other powerful actors, to do the same sort of thing with the next generation of core learning and engineering tools, is freaking terrifying.

[karahime]

I agree, and think the effects on learning should be doubly emphasized. One can lock down everything and everyone to the highest degree possible, think of every possible edge case, set controls 2, 3, 4, 10 steps away from them, but not only is this not beneficial to society overall due to how it hurts adjacent information, it's not even beneficial to the goal in question, since it creates a brittle situation with locks that can't be changed or updated in a world which is always changing and always updating.

[Eji1700]

There are still things considered “state secrets” or similar categories which can very very quickly cause you problems if it’s on a remotely commercial website.

I’m not going to say you can’t find some of this information in shadier spots, but “how do I get my GPS to work on a rocket” or “what kind of math do I need for a fusion implosion” are some of the more extreme examples.

I believe there are several explosive compounds where the formula is decently guarded, although in that case tracking the materials is easier.

I’m not saying anything they’re doing is good, but I feel like since they’re just reinventing the search engine with a lot of this they’re running into similar barriers.

Google has been censoring shit at the whim of governments for years, remotely reasonable or not

[girvo]

The thing is the data isn’t limited, and supply side constraints already solve this problem. I come from a BSc Chemistry background, and they don’t hide how organic chemistry and illegal drug synthesis are intertwined, it’s open information

But where I live the glassware and precursors will get you a very angry knock on the door.

[atemerev]

Well, this is why I had to abliterate GLM5.2 simply out of spite and now I am free to ask all my nuclear weapons design questions I might have.

I really really hate refusals like these.