[lawrjone]

We have found similar when plugging GLM 5.2 into actual benchmarks in our product. The open-source models are really dialled into the public benchmarks, until you try them in context you won't have a solid idea of how they perform (Sonnet is a higher quality model than 5.2, both in prose, reasoning, and alignment).

[ACCount37]

Most benchmarks notoriously undercount multi-turn instruction following too. And it's where open source models (and Gemini) lose big time.

[flumes_whims_]

Probably because benchmarks are leaked. Included in their training or model can cheat by finding answer online.