|
|
|
|
|
|
by utopcell
35 days ago
|
|
|
> You can see which bugs each model found in the full report. This should have probably been surfaced at the text. To answer my own question: If one uses gemma4-26b-a4b, mimo-v2.5-pro and gemini-3.5-flash then 7/9 bugs are covered, while no model can uncover the remaining two. > And, you have misunderstood what the benchmark does. It tells the model to audit the file, and it is allowed to look at the rest of the repo. It is not pointed at the bug. I obviously meant that the model was pointed to the bug report. What would have been the point of providing the actual bug. This is still easier than telling Mythos to generally look at a codebase and find any bug. |
|
|
Many security tools for doing security audits with LLMs are based around a "look at this file" loop, where each file gets analyzed individually. As I noted in the post, I don't consider that a hint at all. In a real security audit, the model would have gotten the prompt "look at this file for security issues".
And, it's probably also how Mythos was used for auditing when it found these bugs. At least a couple of folks at Anthropic have discussed using a loop like that for finding security bugs, which was the inspiration for Nelson, which is what this benchmark project sprung out of.
Nonetheless, I'm currently performing benchmarks of "look at this repo, find any security bugs", because I suspect the really good models will be able to spot some of the hard bugs that span multiple files (the models always have the tools to look at other files, but maybe didn't take time to fully comprehend the full source before tying to find security issues). Those will take a lot longer and cost a lot more. There will be a lot more noise in that benchmark, though, as it'll probably find dozens of real bugs of varying severity and more false positives, which have to be judged, as well.