[notracks]

I recently found out that Claude's latest model, Sonnet 4.6, scores the highest in Bullsh*tBench[0] (Funny name - I know). It's a recent benchmark that measures whether an LLM refuses nonsense or pushes back on bad choices so Claude has definitely gotten better.

[0] - https://petergpt.github.io/bullshit-benchmark/viewer/index.v...

[astrange]

I haven't tried talking to Sonnet much, but Opus 4.6 is very sycophantic. Not in the sense of explicitly always agreeing with you, but its answers strictly conform to the worldview in your questions and don't go outside it or disagree with it.

It _does_ love to explicitly agree with anything it finds in web search though.

(Anthropic tries to fight this by adding a hidden prompt that makes it disagree with you and tell you to go to bed, which doesn't help.)

[sidrag22]

the go to bed thing gets annoying, you can't even hint that you are almost done or wrapping up or something or this is hyper triggered and it never stops.

I do like when opus is incredibly short in its responses to prompts that probably shouldnt have been made though. keeps me grounded a bit.

[layer8]

You don’t have to star out things like that on HN.

[thin_carapace]

it would be interesting to me if you could explain the motivation behind posting your comment. from my perspective, if somebody with 5 years of forum tenure had the intelligence to comment about advanced benchmarks, they probably noticed that censorship was a voluntary decision here, and had made a personal decision on that front.

[mkl]

I'm not layer8, but I had a similar thought. In this case the needless censoring is problematic because it hides the name of the benchmark from future searches (the uncensored URL spells it differently).

[layer8]

Such self-censoring is often done out of habit or a mistakenly assumed obligation to do so. I consider it inappropriate here, as it obscures an actual name, doesn’t constitute an expletive, and the HN readership is generally mature enough to recognize that. The counterquestion is, what justified reason could there possibly be to censor it here? I don’t think there is any, in the sense that people wouldn’t take any offense at the uncensored version, and the intent of my comment was to inform about that.

[notracks]

I censored it out of habit of commenting on other platforms and, I actually didn't have any idea about whether you should censor such words or not in here. Will keep that in mind when commenting here next time.

[thin_carapace]

some people think cursing is bad when done senselessly

[uniq7]

Good call on censoring yourself preemptively, otherwise HN could demonetize your comment

[akurilin]

Great link, thanks for sharing. Confirmed what I saw empirically by comparing the different models during daily use.