[demosthanos]

As someone on the company response side of the HackerOne brokenness, I can confirm that this effect is real but would also note that the difficulty of distinguishing is not as severe as all that, because the companies have access to the source code which the researchers do not typically have access to.

This means that the token cost of verifying any given HackerOne report is dramatically lower than the token cost of producing a report in the first place. Automated triage systems should be possible, and realistically it's well within the capabilities of most companies to go further and actually automate the Red Team side of it and catch issues before they surface in the black box research. From what I've seen doing so should cost dramatically less in tokens than the bounty payouts do.

The problem is that security is woefully underfunded in most companies, so even an infosec organization that saw the deluge approaching from a distance may well not have had the resources to prep for it even if they knew exactly what actions they would take if they had the capacity.

[QuadmasterXLII]

The token cost of a report is lower bounded by the number of tokens in the report * price per token of the cheapest model. The token cost of a good report is much higher, but sifting out the good reports is the entire problem.

[demosthanos]

In theory, yes, but are you actually seeing clearly-garbage lower-bound reports like this?

The ones we're seeing show clear evidence of being AI-generated, are often incorrect or duplicated, but they also show clear evidence of the AI having done its homework and spent a while crawling our API.

Even if we were getting reports at the lower bound you're describing, those would be even easier to triage: just add a quick step to check if the API in question even exists, then if it does that very cheap "where is this API" query becomes part of the input to the second-level triage that spends more tokens.

[infinite_spin]

Do you have a tier based system for identifying high value reporters? Like a credit score for vulnerability hunters?

[demosthanos]

Not formally, but we do informally recognize names and prioritize researchers who reliably turn up good results.

[infinite_spin]

that might be a nice incentive for bounty hunters, a sign of recognition and a good metric for triage