[ryeights]

Yes, but Anthropic didn’t already know the answers. In the OSS ‘reproductions’, they fed the model the one file that actually has a vuln and even told it which parts of the code to focus on. This is obviously a much easier task.

If OSS models are equally up to the task, why not find novel vulnerabilities?

[firer]

Yeah, totally agree now that I've looked into it more.

> If OSS models are equally up to the task, why not find novel vulnerabilities?

To be fair, in the same blog post Anthropic mentioned costs in the tens of thousands of dollars per project looked at it. So it's a big ask to do an experiment that compares. Would love to see it though.