Hacker News new | ask | show | jobs
by wgd 23 days ago
"agent pipelines that [...] clean a messy [repository]"

This feels like a terrible approach, sufficient to condemn the entire study.

Apparently half of the "minimal pairs" in this work were constructed in this way. I simply am not going to trust any conclusion that requires assuming these AI "cleaned" repos are in any way representative of actually-good codebases.
2 comments
geraltofrivia 23 days ago
First author here. Please let me offer a clarification. Our notion of "clean" isn't to just ask the agent to write better code. Rather, we give it a list of 50-100s static analyzer rule violations (and code LOC), and ask to remove them. We then check if the rule violations are resolved.

Using LLMs to rewrite code to remove these violations is a rather accepted practice. Sonar's existing one-shot LLM based approach [1] (in production since 1+ year), and a recent agentic approach [2] to do the same work rather well to do this.

[1] https://www.sonarsource.com/solutions/ai/ai-codefix/

[2] https://www.sonarsource.com/products/sonarqube/remediation-a...

link
CuriousSkeptic 22 days ago
I do wonder though, how does metric based factoring compare to expert intuition based factoring? Can the latter be emulated by agents? Are there studies?
link
geraltofrivia 16 days ago
Not to my knowledge :)
link
ramraj07 23 days ago
Would you trust clean repos that are messed up by AI?
link
wgd 23 days ago
Yes, those ones would be at least a somewhat-plausible simulation of a real scenario people care about: a once-clean codebase that was allowed to become messy by a succession of insufficiently-careful vibeslop PRs.

I'm not a huge fan of their methodology for the AI-degraded cases either (ideally one would set up the mirror pairs by taking some real repositories and rewinding history a month or so and then having a succession of independent agents reimplement each bit of feature work and bugfixes over that period of time), but it's at least a coarse approximation whereas I just don't trust the cleanup methodology to resemble anything real in the first place.

link
geraltofrivia 23 days ago
Our initial suggestion was to do something along the lines of your proposal. But we found that when we ask an agent to implement features for `t-n`^th PR, even when we are overly specific makes the code rather divergent, to the point that sometimes the `t-n+1`th task description doesn't really make a lot of sense.

In fact, there are some papers (that we cited) which create a set of tasks doing exactly this, and it is non-trivial [1].

[1] https://arxiv.org/html/2603.24755

link
smallerize 23 days ago
No, because the outputs will still be "in distribution" so to speak.
link