[m132]

A hard read for a skeptic like me. A lot of speculation and extrapolation of a trend, not to say outright exaggeration, but very little actual data. Let's not forget that we're at the tip of an economic bubble, and what you're writing about is at the very center of it!

For what it's worth, I read Anthropic's write-up of their recent 0-day hunt that most of this post seems to be based on, and I can't help but notice that (assuming the documented cases were the most "spectacular") their current models mostly "pattern-matched" their ways towards the exploits; in all documented cases, the actual code analysis failed and the agents redeemed themselves by looking for known-vulnerable patterns they extracted from the change history or common language pitfalls. So, most of the findings, if not all, were results of rescanning the entire codebase for prior art. The corporate approach to security, just a little more automated.

Hence I agree with "the smartest vulnerability researcher" mentioned near the end. Yes, the most impactful vulnerabilities tend to be the boring ones, and catching those fast will make a big difference, but vulnerability research is far from cooked. If anything, it will get much more interesting.

[acdha]

I tend to be skeptical but listening to the linked podcast with Carlini and found him very credible–not a sales guy, not an AI doomer, but someone talking about how little work he had to do to find real exploits in heavily-fuzzed code. I think there’s still a safe bet that many apps will be cumbersome to attack but I think it’s still going to happen faster than I used to think.

https://securitycryptographywhatever.com/2026/03/25/ai-bug-f...

[tptacek]

Nicholas Carlini is the real deal. He was most recently on the front page for "How to win a best paper award", about his experience winning a series of awards at Big 4 academic security conferences, mostly recently for work he coauthored with Adi Shamir (I'm just namedropping the obvious name) on stealing the weights from deep neural networks. Before all that (and before he got his doctorate), he and Hans Nielsen wrote the back half of Microcorruption.

He's not a sales guy.

[acdha]

Thanks for having him on. It was really nice to hear a sober, experienced voice talking about their work with fellow practitioners.

[tptacek]

Thank Nicholas! We'll talk to anyone. :)

[m132]

Thanks. Watched most of this talk and, unless I missed something, it seems to confirm what I was thinking—most of the strength currently comes from the scale you can deploy LLMs at, not them being better at vulnerability research than humans (if you factor out the throughput). And since this is a relatively new development, nobody really knows right now if this is going to have a greater impact than fuzzers and static analyzers had, or if newer models are ever going to get to a level that'd make computer security a solved problem.

[woeirua]

Theres a video of a recent talk Nicolas Carlini gave this past week on Youtube. It’s eye opening. If you don’t believe that LLMs are going to transform the cybersecurity space after watching that I can’t help you.

[tptacek]

It's this talk right here:

https://www.youtube.com/watch?v=1sd26pWhfmg

7 minutes in, he shows the SQLI he found in Ghost (the first sev:hi in the history of the project). If I'd remembered better, I would have mentioned in the post:

* it's a blind SQL injection

* Claude Code wrote an exploit for it. Not a POC. An exploit.

[streetfighter64]

> Not a POC. An exploit.

What's the distinction? A proof of concept is just something that demonstrates that a bug is possible to exploit, by doing so.

[cushychicken]

Repeatability and/or an actual negative effect.

POC generally means “you can demonstrate unintentional behavior”.

“Exploit” means you can gain access or do something malicious.

It’s a fine line. Author’s point is that the LLM was able to demonstrate some malfeasance, not just unintended consequence. That’s a big deal considering that actual malicious intent generally requires more knowhow than raw POC.

[tptacek]

Specifically: the exploit extracted the admin's credentials from the database. A blind SQLI POC would simply demonstrate the existence of a timing channel based on a pathological input.

[cushychicken]

One other commenter asked a decent question - does going lighter (Zig) or harder on memory safety (Rust) confer any meaningful advantages against the phenomenon you describe?