[_0ffh]

The question is: If biological, computer security, and ML research are so bad, why do they even train on the relevant data?

The only answer that makes sense is they wanted the model to be competent and usable in these fields, just not by you, which is why they had to bolt on a badly functioning crippling device after the fact.

[sweetjuly]

Is what you suggest about training even possible? Most exploitation techniques are really just about having in-depth knowledge of how components work. For example, I imagine a sufficiently powerful model could fairly easily re-invent the ROP chain from first principles if it just knew how the stack works. This same principle applies to much more complex attack too; exploitation is often just an exercise in knowing vastly too much trivia, which LLMs tend to have in spades.

[_0ffh]

It would still degrade it's effectiveness, which is what they claim to want. Exaggeratedly: If it wasn't so, you'd just need fundamental math in the training data, as everything else can be derived.

[ACCount37]

Remove the relevant data, and just enough of the data around it will remain that the AI will be able to close the gap if given relevant documentation.

Not to mention that those capabilities are inherently dual use. If you know how to write C safely, you know how to spot unsafe C.

[solenoid0937]

Or they wanted the model to be good at these things, for the companies that legitimately need access to these capabilities.

[siva7]

so only the chosen for-profit companies by Anthropic are allowed to use frontier ai in the name of safety? what kind of joke is that? you people here can't be that dumb..

[solenoid0937]

how is that dumb? Should every random company/person be allowed to develop cyber weapons or bio weapons?