[faitswulff]

> The analysis uses a single metric: bugs per 10 commits (bugs/10c).

Bugs per commit as a metric papers over severity, both in terms of security severity as well as the effect on the user. A mislabeled button has the same weight as the entire app crashing in this framework.

[germanjoey]

IMO "bugs per commit" is even worse than that, because, in addition to what you say, it also hides the extraordinary spike of commit activity of a project that had previously been stable. [0]

It is the exact metric you'd choose if you wanted to make the current situation of rsync look like not a big deal.

[0] https://github.com/RsyncProject/rsync/graphs/commit-activity

[logicprog]

Yes, but we know why there was an "extraordinary spike," and it has nothing to do with rsync being "vibe coded." The maintained has directly addressed this.

[vsundar]

> The maintained has directly addressed this.

Not sure if this is mentioned somewhere else, but looks like the maintainer has a blog post that addresses this: https://medium.com/@tridge60/rsync-and-outrage-d9849599e5a0

[floxy]

Seems like this would be a good place to link to that.

[logicprog]

I link to it multiple times in TFA and quote the specific thing I'm talking about here in there to explain that possible confounder. I think I've done more than the work I'm obligated to it.do to make all of the relevant information available to you. You are just refusing to use

[runarberg]

I am not finding these links in TFA, I see a link to an issue #929 which (as mentioned in TFA) has over 350 replies, and and opinionated summary of what transpired, including some detailed description of specific posts there. However I did not find the maintainers response.

Of interest is this post here: https://github.com/RsyncProject/rsync/issues/929#issuecommen... which echos the same concern which was raised up thread, however, I failed to find the maintainers’ response.

EDIT: Found it! it is in the (untitled) discussion section (after the results).

https://lobste.rs/s/k1b0za/rsync_outrage#c_2iowov

EDIT 2 (and advice on design): The page design changes backgrounds after the results sections, which kind of conveys to the user that they have reached the end of what was is important and can just skim over the rest (usually pages have a radical change in typography like these when you’ve reached the comment section), however this is what is analogous to a discussion in a typical paper, and is arguably the most important part. I had simply assumed that you just left it at the result and skipped the discussion as a stylistic choice.

[logicprog]

> EDIT: Found it! it is in the (untitled) discussion section (after the results).

I also paraphrase Tridge himself explicitly saying that this is why commits/releases have increased:

> Essentially, this isn't a "Claude" problem, it's a "more security work" problem, something that Tridge himself confirmed in his response, describing how a flood of AI-generated CVE reports forced rapid, extensive changes to rsync's attack surface.

> The page design changes backgrounds after the results sections, which kind of conveys to the user that they have reached the end of what was is important and can just skim over the rest (usually pages have a radical change in typography like these when you’ve reached the comment section), however this is what is analogous to a discussion in a typical paper, and is arguably the most important part. I had simply assumed that you just left it at the result and skipped the discussion as a stylistic choice.

Good point, I assumed everyone would read till the end, that's on me. I'll give it a heading.

[ex-aws-dude]

Why don't you prove the bugs increased then?

Why is it that some unfounded claim is made and the onus is suddenly on the project maintainer to prove it beyond all doubt?

It should be on the person making the claim to prove it

[logicprog]

I've now resolved this. The new version, which should be live on GH Pages soon, uses — what I think is — a pretty good methodology for assigning severity to each bug, normalizes it to 0.0-1.0, sums that, and treats that as the total severity weighted bugs, then does the analysis based on that. It did not change the analysis in any material way.

[bsza]

No Claude, it still makes zero sense as a metric.

A commit is a measure of nothing. Severity weighted bugs per unit of nothing? What does that even mean? In any repo it's trivial to achieve a sev/10c that's arbitrarily close to zero while completely ruining everything.

I suggest you practice some humility and update your conclusion instead of updating the mental gymnastics you used to arrive at the same conclusion.

[logicprog]

Whether commits decrease the sev/10c depends on if there are a lot of small commits increasing the demoninator. In reality, we have the opposite: the post-Claude releases have way fewer commits than the pre-Claude ones.

Thus, if anything their sev/10c is inflated. If I changed it to lines of code changed, the relative bug ratios would be much smaller, and the conclusion wouldn't change. In fact, the conclusion would look "better" for Claude; if I was using "mental gymnastics" to come to this conclusion, I would have already used a metric other than adjusting per commits!

What different metric would you suggest that would change the conclusion?

Showing "humility", as you so moralistically and condescendingly put it, would require being wrong first.

[bsza]

> In reality, we have the opposite: the post-Claude releases have way fewer commits than the pre-Claude ones.

No, they don't, you just made that up.

> What different metric would you suggest that would change the conclusion?

What would be a lot more useful to know is whether or not the original prompt used to generate this post instructed you to do a fair and unbiased review of these bugs, and whether or not that prompt itself was framed in a fair and unbiased way. If you take a piece of paper and write "therefore Claude is not at fault" at the bottom, then nothing you write above that line is admissible, no matter how well-reasoned.

[skeledrew]

There was no analysis of severity in all of the rage posting that occurred. The single point being pushed was "use of an LLM led/leads to more bugs". The author specifically states that's what they're addressing (blunt accusation -> blunt response).

[atmavatar]

The specific problems mentioned were all reasonably severe. The original post itself described a show-stopping bug:

    So my systems recently updated to rsync 3.4.3, and as soon as that happened my backup system - which does incremental backups using multiple --compare-dest= arguments - started to fail on anything but a full backup.

Incremental backups is perhaps the primary use of rsync, and they were broken for this person. That's pretty severe.

The second reply is similar:

    i wondered why my 3d printers were running like sh*t and at 100% cpu; turns out log2ram uses rsync.

This one I took with a grain of salt, since it read more like a dogpile than an actual bug report. However, if it's genuine, it's also reasonably severe.

Later in the comments, someone attempted to provide a list of issues that had been added: https://github.com/RsyncProject/rsync/issues/929#issuecommen.... The list included several failures to build or run rsync that appear to have resulted from broken backward compatibility. That seems reasonably severe. If intentional, I would have expected mention in the release notes about the removal of backwards compatibility, but none was made.

The issue comments already degraded into a lot of unnecessary vitriol even before the above mentioned comment and only gets worse from there, so I stopped. But, the fact remains that the whole issue started with a severe bug.

I applaud the attempt at dispassionately analyzing whether the recent LLM releases of rsync were normal or outliers as far as bugs are concerned, but I don't think you can do so properly without analyzing severity.

[skeledrew]

To keep such an analysis fair and contextually relevant, it would have to be extended to the previous 928 issues as well (of course filtering for bug reports). I don't see anyone doing such an analysis, I think because they don't expect they'd find it useful (at least not as the rage fuel that many are seeking); what they'd be more likely to find is that there is a similar severity-mix going all the way back to v1.0.0, because these things inevitably happen whether coding is done by human or machine.

"A lot of claims in the wider discussion have treated every recent bug report as if it had the same cause. That is not accurate. Some reports were regressions from recent security hardening, some were missing historical test coverage, some were older bugs found because rsync suddenly had more eyes on it (especially by AI that can find issues quickly) and some were packaging or environment-specific failures. A Co-authored-by line is not enough by itself to establish root cause." - https://github.com/RsyncProject/rsync/issues/929#issuecommen...