[codedokode]

Sadly no tasks for C, especially for working with optimized low-level data structures. And with testing performance of a solution. Anyone can write Python, try writing optimized low-level code.

Also I wonder if models playing dumb to prevent learning on outputs affected the score.

Also interesting that Claude edits files by writing and running Python scripts, is that efficient?

[Topfi]

I just checked and for plain old C, there do not seem to be any reasonably comprehensive, current-day eval suites. Fully admitting that, even if there were, I couldn't assess their validity simply because I have never written or reviewed any C code in my life (something I should rectify probably). Maybe the closest proxy is just parsing through the experiences people claim to have whenever LLM assisted kernel development comes up [0], but if you have a dataset, experience, time and muse, I'd just go for it and do some tests yourself. Have been doing the same, mainly focused on code quality and dealing with a mix of Rust, frontend web tech and SQL which has been a small but meaningful project and part of my go to eval for over a year now.

I doubt that, in these tasks, model restrictions to prevent training are affecting the results, not least because for both evals, the labs provided pre-release model access and have an incentive to be seen as favorably. In any case, I have not seen regressions to prevent distillations myself even when working on microscopic model training projects with LLM assistance, what I have however reliably and consistently seen is that some providers do train on popular evals and can underperform with minor changes to the task due to that.

Yes, harnesses, including Claude Code can prompt the models to write throwaway code to execute certain tasks, mostly Python, bash scripts or TS/JS, with there being some biases towards one over the other depending on the lab or specific model. Mainly for repetitive tool calls with no pre-existing/provided tools enabling it. Is in most instances a lot more efficient then a model e.g. doing a refactor that requires consistent variable renaming directly and around Opus 4.1/GPT-5, models have been trained to very consistently and accurately gauge when a task can benefit from such scratchpad scripts vs when that is inefficient/not useful.

[0] https://news.ycombinator.com/item?id=44990981

[codedokode]

Also, GLM 5.2 seems to be the best open-weight model, and it beats proprietary Gemini and older versions of Claude which is amazing. You can have a model at level of Claude Sonnet 4.6 at home without sharing anything, and maybe even uncensor it.