[bastawhiz]

If you're measuring a sloppy project against a well structured one with regard to how many tests fail, that'll bias success towards the sloppy codebase, which likely has worse test coverage and less robust testing in the first place. You'd essentially need to write a single test suite that works for both projects in each pair to compare fairly.

That's not to say the study is good, but I can respect their decision because the tests passing isn't necessarily correlated with the effectiveness of the agent.

[geraltofrivia]

> You'd essentially need to write a single test suite that works for both projects in each pair to compare fairly.

That's pretty much what we did. We start with a repository, and

- (vibeclean pipeline) make changes to clean analyzer issues (some of them involve moving big code chunks around to reduce cognitive complexity), and tweak the existing test sets when needed.

- (slopify pipeline) inverse, add analyzer issues and increase complexity, add dead code etc.

In both cases, we ensure that the test coverage remains the same, and the tests are passing, before we start our experiments.