[yourapostasy]

In short, no.

The ArXiv paper mentions the human developer must supply a unit test (which can conceivably be coded with at least the assistance of an AI agent if not autonomously coded, but their experiment relies upon the former kind of unit test) that issues a pass-fail signal. So the 78% of failures are clearly identified, at the cost of implementing TDD for the Issue. The side effects story is punted upon, but I’d still take this over the nothing we have today.

Of course, over a relatively short amount of time using this, I’d expect to experience the 22% (or whatever the real rate is) success rate to drop asymptotically towards zero as the low hanging fruit of the approach are mined out and it becomes kind of like another linter in our CICD pipelines.

The impact of this tooling upon staff skills development will be interesting to say the least.

[yuntong]

AutoCodeRover does not require or assume a unit test to generate patches. The results discussed in Section 6.1 of the ArXiv paper are generated without any unit test. The unit tests are used by SWE-bench, when evaluating the correctness of AutoCodeRover-generated patches.

That being said, when some unit tests are available (either written by developers or with assistance from other tools), AutoCodeRover can make use of them to perform some analysis like Spectrum-based Fault Localization (SBFL). This kind of analysis output can help the agent in pinpointing locations to fix. (Please see Section 6.2 for the analysis on SBFL.)

[dboreham]

> AutoCodeRover does not require or assume a unit test to generate patches.

You have this backwards : it's traditional (at least in the past 15 years or so) to have a test to go along with every code change. The idea is that the test proves a) the bug existed prior to the fix and b) the bug is not there after the fix is applied. Commenters here are noting that ACR generates fixes but not tests.

[yuntong]

The previous comment was to describe the experiment settings. AutoCodeRover currently generates patches. Auto-generating high quality tests can be a parallel effort and another direction to explore. These efforts can eventually be used together.

[dboreham]

The point is that a patch without a test is not generally a useful thing. How do we know the AI generated patches are valid?

[abhikrc]

We know that by testing the patches. Automated Program Repair is an area of research which has done this.

The patches are tested against a test-suite. So, if there are tests, we welcome them, we definitely use them to validate the patches.

[nfm]

I agree in principle, but if it also generated a test, how would you know that was valid?

The value I get from copilot is the ability to code faster, not the ability to code.

[abhikrc]

The short answer is that unit tests are not needed in AutoCodeRover. The technique proceeds by a sophisticated code search starting from the Github issue. tests are not needed. The code search helps in setting the context for LLM agents - which can help in the patch construction.

If tests are available, they can give additional help in setting code context. But tests are not needed, and most of Github issues are solved without tests.

All experimental numbers appear in the arxiv paper. Please let us know if you have more questions.

[dboreham]

> tests are not needed

Strong words!