[WilcoKruijer]

I hope the big labs will start using this benchmark in their RL pipelines. Reducing complexity in generated code should be the number 1 priority, in my opinion. The holy grail for me is models implementing features while reducing LoC (i.e., choosing the right abstractions).

What is also nice about this benchmark is that it can be used to iterate on prompts/skills for reducing code complexity.

[tracker1]

I call out specifically in my agent/claude file that I want and prefer simpler solutions over complex enterprise pattern usage and overuse of abstractions. It tends to help a bit, but often I have to give feedback in my review step.

My typical workflow when AI assisted is what I call human gatekeeping... I'll plan out next steps with the agent, updating a TODO.md file with what needs to be done, then a fresh context to implement the next step(s), and review the code before committing to git. I may iterate/stash or even reset a few times before it's "good enough"... it is rarely close to what I would do myself, but often as good as what I've gotten from other developers IRL on projects.

This includes updating the documentation area(s) of a project as well as testing. I also tend to lean on ./run/* for scripts that will run/test various portions of the project... getting the agent to use these is sometimes harder than it should be, as I want it to specifically exercise a lot of things through the process... there are also times where it will try to change a valid test that's failing instead of fixing the code. That's the most irritating part. Some models are annoying, some feel like pure magic at times.

[falsemyrmidon]

It's just as easy to go too far in the direction of shoving too much logic on a single line in the name of reducing lines of code.

[Normal_gaussian]

It should be hoped that whoever is doing this is using automatic formatters; though of course metrics relating to code complexity are best done as measures over the AST.

[blcknight]

> I hope the big labs will start using this benchmark in their RL pipelines.

Labs do not train on benchmark data (allegedly). They can train on similar problems, but benchmarks have specific strings in them that labs are supposed to be aggressive in filtering out of their training corpora.

[jerf]

I would at least consider the possibility that they already are, and this is how it is going. It could be a fundamental architecture problem for LLMs. It's not like "slop" is a new problem and I'm sure they'd love to announce a new model that generates much less "slop". The fact they've never so much as mentioned it suggests to me that it's not something they've been able to fix.