Hacker News new | ask | show | jobs
by shevy-java 110 days ago
I guess it is good when bugs are fixed, but are these real bugs or contrived ones? Is anyone doing quality assessment of the bugs here?

I think it was curl that closed its bug bounty program due to AI spam.

2 comments

The bugs are at least of the same quality as our internal fuzzing bugs. They are either crashes or assertion failures, both of these are considered bugs by us. But they have of course a varying value. Not every single assertion failure is ultimately a high impact bug, some of these don't have an impact on the user at all - the same applies to fuzzing bugs though, there is really no difference here. And ultimately we want to fix all of these because assertions have the potential to find very complex bugs, but only if you keep your software "clean" wrt to assertion failures.

The curl situation was completely different because as far as I know, these bugs were not filed with actual testcases. They were purely static bugs and those kinds of reports eat up a lot of valuable resources in order to validate.

The bugs that were issued CVEs (the Anthropic blog post says there were 22) were all real security bugs.

The level of AI spam for Firefox security submissions is a lot lower than the curl people have described. I'm not sure why that is. Maybe the size of the code base and the higher bar to submitting issues plays a role.

The issue is what each of the projects considers viable bug if you consider all localized assertion failures possible bugs then that's different from give me something that practically affects users.

Further browsers have a much larger surface area for even minor fuzzing bugs. Curl's much smaller surface area is already well fuzzed and tested.

Chrome has better fuzzing and tests too. Firefox has had fewer resources compared to Google ofc, so understable.

Ofc not saying it wasn't good. But given the LLM costs I find it hard believe it was worth it, compared to just better and more innovative fuzzing which would possibly scale better.