[narginal]

Just like how fuzzers will find all the bugs, right? Right?? There's definitely infrastructure at these big companies that isn't sitting in a while loop 'fuzzing' right? Why is it news that vulnerability research will continue to get harder, exactly? It has always been this way, exploits will get more expensive, and the best researchers will continue with whatever tools they find useful.

[tptacek]

It's a good question. Fuzzers generated a surge of new vulnerabilities, especially after institutional fuzzing clusters got stood up, and after we converged on coverage-guided fuzzers like AFL. We then got to a stable equilibrium, a new floor, such that vulnerability research & discovery doesn't look that drastically different after fuzzing as before.

Two things to notice:

* First, fuzzers also generated and continue to generate large stacks of unverified crashers, such that you can go to archives of syzkaller crashes and find crashers that actually work. My contention is that models are not just going to produce hypothetical vulnerabilities, but also working exploits.

* Second, the mechanism 4.6 and Codex are using to find these vulnerabilities is nothing like that of a fuzzer. A fuzzer doesn't "know" it's found a vulnerability; it's a simple stimulus/response test (sequence goes in, crash does/doesn't come out). Most crashers aren't exploitable.

Models can use fuzzers to find stuff, and I'm surprised that (at least for Anthropic's Red Team) that's not how they're doing it yet. But at least as I understand it, that's generally not what they're doing. It something much closer to static analysis.

[staticassertion]

I suspect we'll see combinations of symbolic execution + fuzzing as contextual inputs to LLMs, with LLMs delegating highly directed tasks to these external tools that are radically faster at exploring a space with the LLM guiding based on its own semantic understanding of the code.

I'm with you, I expected this to be happening already. Funny enough, I guess even a hardened codebase isn't at that level of "we need to optimize this" currently so you can just throw tokens at the problem.

[tptacek]

Right, so that's exactly how I was thinking about it before I talked to Carlini. Then I talked to Carlini for the SCW podcast. Then I wrote this piece.

I don't know that I'm ready to say that the frontier of vulnerability research with agents is modeling, fuzzing, and analysis (orchestrated by an agent). It may very well be that the models themselves stay ahead of this for quite some time.

That would be a super interesting result, and it's the result I'm writing about here.

[narginal]

I have just seen too much infrastructure set up to 'find bugs,' effectively sitting and doing nothing- either the wrong thing gets audited, or tons of compute gets thrown at a code base and nobody ever checks in on or verifies.

This seems like a human/structural issue that an AI won't actually fix - attackers/defenders alike will gain access to the same models, feels a little bit like we are back to square one

[tptacek]

If that's true, and if patches can effectively be pushed out quickly, then the results of this will be felt mostly by vulnerability researchers, which is the subject of the piece. But those are big "ifs".

[gsnedders]

The other thing is to remember is that when it comes to complex targets, attacks still are found by using a different fuzzer and/or targeting a different entry point.

It stands to reason that the same will apply for LLMs.