[tempestn]

> If Claude gives me poor or incorrect advice while I’m working on an AI component, I have no way of knowing whether the model was confused, whether my problem is unsolvable, or if some invisible policy restriction quietly kicked in.

You should be able to know if your problem was solvable by using your own expertise and judgement, no? If you're relying on LLMs as a substitute for those, I wouldn't expect great results.

[notrealyme123]

You come up with a hypothesis -> you let fable implement it -> fable sabotages your experiment -> you get evidence that hypothesis is not true.

It's that simple.

[hedora]

Or, worse:

- It says your safety hypothesis is true, you incorrectly ship, killing lots of people.

- It proposes dangerous experiments.

[hedora]

No; once the LLM switches to this new saboteur mode, it’ll be very hard to detect.

Sabotage is an asymmetric weapon. The ratio of damage to effort is nearly unbounded, and any decent saboteur knows that the key trick is to make your output indistinguishable from incompetence.

They’re building state of the art offensive capabilities into a public model, then expecting to maintain control over when it decides to attack its human users.

The premise is laughable, and we’ve all seen how this movie ends.

[HarHarVeryFunny]

Great way for Anthropic to build trust with the military