[somesortofthing]

Am I wrong in thinking that an "exploits are free" environment massively favors the defender? Given that real-world exploits usually chain 0days, the attacker has to find the whole chain while the defender only needs to fix the weakest link.

The defender also gets to make the first move by just putting a "run an agent to find vulns" step in their CI pipeline. If LLMs truly make finding exploits free, almost no LLM-findable exploits will ever make it into the codebase.

The only way break the equilibrium is still going to be a smart researcher capable of finding exploits that the commoditized tools alone can't.

[ActorNightly]

You aren't wrong for the most part, but this whole thing of "find me vulns" is not really accurately describing the space.

Finding vulns has almost become sort of like a vibe thing even before LLMs. There would be some security patch that everyone says is critical because it fixes a vulnerability, but the vulnerability is like "under certain conditions, and given physical access to the device, an attacker can craft a special input that crashes the service"... and thats it.

Even stuff like Spectre and Meltdown, which I highly doubt an LLM can find on its own without specifically knowing about speculative execution attacks, are incredibly hard to use. People made a big deal of those being able to be used from javascript, but to actually leak anything of importance you need to know memory layouts, a bunch of other info and so on.

So while an LLM can patch all the up front vulnerabilities, most if not all of those are completely useless to an attacker. Modern systems are incredibly secure.

On the flip side, the stuff that LLM doesn't know about, that can be exploited. For example, assume that log4shell hasn't been found yet, and that log statements by default can pull jni objects from the internet and execute them. The llms would happily write you code with log statements using log4j, and pass it through vulnerability checker, and I would bet that even at the bytecode level it won't ever figure out that vulnerability exists.

And overall, because of Rice theorem, you can't tell if the program is fully exploitable or not without actually running it in some form and way. LLMS can help you with this (but not fully of course) by actually running it and fuzzing inputs and observing memory traces, but even this gets very hard when you introduce things like threading and timed executions which can all affect the result.

And also, the LLMs themselves are an exploit vector now. If you manage to intercept the API calls somehow and insert code or other instruction, you can have the developer essentially put the exploit for you into the code.

So I would say the field is about even.

[bawolff]

> Even stuff like Spectre and Meltdown, which I highly doubt an LLM can find on its own without specifically knowing about speculative execution attacks, are incredibly hard to use. People made a big deal of those being able to be used from javascript, but to actually leak anything of importance you need to know memory layouts, a bunch of other info and so on.

In fairness, i think part of the reason people made a big deal was the novelty of the attack. It was something new. The defenses weren't clear yet. The attack surface wasn't clear. It was unclear if anyone was going to come up with a novel improvement to the technique. Humans love novelty.

[pron]

Right, the article makes it seem that the attackers have the advantage in that they can try to attack everything at once, but the defenders have the advantage in that they each need to defend only one product.

But it is true that if attackers more easily find holes in software that's hard to patch (embedded), that's a problem.

[benlivengood]

The economics favor attackers. Who sells 0-days for quite a lot of money (or directly exploits them for ransomware), vs. who has to pay engineers quite a lot of money to review bug reports and patch code and publish new releases?

The validation/verification balance also favors attackers. "Yes, I now have a remote root shell on this VM with a default install of X" vs. "My test suite is not dependable enough to turn an agent loose fixing security bug reports, not to mention the extra QA work that live humans would have to do where there isn't coverage".

[torginus]

Yes I agree. After a period of turmoil, there'll be a new equilibrium between attackers and defenders.

Considering the new 'meta' that LLMs encode knowledge about existing software but not new ones, I would expect a side effect that newly written software will be inherently less exploitable by LLMs, even if from an actual security perspective, they're worse in design.

[dandanua]

The world runs on legacy software, which simply has no budget for even asking LLMs to find vuls, not to mention fixing them. While you are right that it should favor the defenders, in practice, this is a disaster because, in many critical cases, there are no defenders.

[lukewarm707]

there are tools already like trufflehog that will find leaked secrets

macos still ships with live github private keys in the library folder

oops i put my god level aws key in js on my website

oops i put the supabase key in frontend

oh no! i am the maintainer of a hosted password manager and i took home the cmek key

[strogonoff]

The higher the productivity multiplier towards exploiting software, the more developers would find themselves severely outmatched: exploiting software is someone’s full-time job, whereas the engineers already have one—building it.

To express this in numerical terms, let’s consider developer’s incentive to spend effort learning to find and actually finding vulnerabilities in their software (as oppposed to building it) as D, and attacker’s incentive to spend effort exploiting that software as A.

I would say initially A = D × 5 is fair. On one hand, the developer knows their code better. However, their code is open, and most software engineers by definition prefer building (otherwise they would have been pentesters) so that’s where most of their time is going. This is not news, of course, and has been so since forever. The newer factor is attackers working for nation-states, being protected by them, and potentially having figurative guns to their heads or at least livelihoods depending on the amount of damage they can deal; the lack of equivalent pressure on the developer’s side leads me to adjust it to A = D × 10.

×10 is our initial power differential between the attacker and the developer.

Now, let’s multiply that effort by a constant L, reflecting the productivity boost from LLMs. Let’s make it a 10 (I’m sure many would say LLMs make them more tham ×10 more productive in exploit-finding, but let’s be conservative).

Additionally, let’s multiply that by a variable DS/AS that reflects developer’s/attacker’s skill at using LLMs in such particular ways that find the most serious vulnerabilities. As a random guess, let’s say AS = DS × 5, as the attacker would have been exclusively using LLMs for this purpose.

With these numbers substituted in, X would be our new power differential:

    X = (A × L × AS) ÷ (D × L × DS)

    X = (D × 10 × 10 × DS × 5) ÷ (D × 10 × DS)

    X = 50. 

If my math is right, the power differential between the attacker and a developer jumps from 10 to 50 in favour of the attacker. If LLMs ×100 the productivity, the new differential would be 500.

I didn’t account for the fact that many (especially smaller) developers may not even have the resources to run the equivalent compute power as a dedicated hacking team.

Some ways to shift the balance back could be ditching the OSS model and going all-in on the so-called “trusted computing”. Both measures would increase the amount of effort (compute) the attacker may need to spend, but both happen to be highly unpopular as they put more and more power and control in the hand of the corporations that build our computers. In this way, the rise of LLMs certainly advances their interests.

[pron]

> exploiting software is someone’s full-time job, whereas the engineers already have one—building it.

But the attackers needs to spread their attack over many products, while the engineers only need to defend one.

> The newer factor is attackers working for nation-states, being protected by them, and potentially having figurative guns to their heads or at least livelihoods depending on the amount of damage they can deal; the lack of equivalent pressure on the developer’s side leads me to adjust it to A = D × 10.

Except that's true even without LLMs. LLMs improve both sides' capabilities by the same factor (at least hypothetically).

> Additionally, let’s multiply that by a variable DS/AS that reflects developer’s/attacker’s skill at using LLMs in such particular ways that find the most serious vulnerabilities. As a random guess, let’s say AS = DS × 5, as the attacker would have been exclusively using LLMs for this purpose.

I'm not sure that's right, because once attackers develop some skill, that skill could spread to all defenders through tools with the skill built into them. So again, we can remove the "LLM factor" from both sides of the equation. If anything, security skills can spread more easily to defenders with LLM because without LLMs, the security skill of the attackers require more effort to develop.

[strogonoff]

> > exploiting software is someone’s full-time job, whereas the engineers already have one—building it.

> But the attackers needs to spread their attack over many products, while the engineers only need to defend one.

Are you assuming every piece of software has a dedicated defender team? Strikes me as unlikely.

Realistically, you have people whose job or passion is to develop software, who often work not on one but on N projects at the same time (especially in OSS), and who definitely aren’t going to make finding vulnerabilities their full-time job because if they do then there’ll be no one to build the thing in the first place.

> Except that's true even without LLMs.

Of course. That’s why I put it before I started taking into account LLMs. LLMs multiply the pre-existing imbalance.

> once attackers develop some skill, that skill could spread to all defenders through tools with the skill built into them

Sure, that’s an interesting point. I’m sure the attackers try to conceal their methods; the way we tend to find out about it is when an exploit is exhausted, stops being worth $xxxxxxxx, and starts to be sold on mass markets, at which point arguably it’s a bit late. Furthermore, you still mention those mystical “defenders”, as if you would expect an average software project to have any dedicated defenders.

(Edited my reply to the latest point, I didn’t read it correctly the first time.)

[pron]

> Are you assuming every piece of software has a dedicated defender team? Strikes me as unlikely.

No, I'm assuming it has maintainers (they play the role of defenders).

> engineers who work on software are simply not that great and dedicated about finding vulnerabilities in it.

Yes, but LLMs help them more than they help the attackers, because the attackers are already security experts. In other words, the LLMs reduce the skill gap rather than increase it. Becoming good at using AI is much easier than becoming good at security.

[strogonoff]

> I'm assuming it has maintainers (they play the role of defenders).

A maintainer has a full-time job: to develop software. A maintainer who is also a defender has two full-time jobs, and as we all know in such a case one of these jobs will have to be done poorly, and we all know which one that is.

On the other side there’s an attacker with a singular job and a strong incentive to do it well.

> LLMs help them more than they help the attackers, because the attackers are already security experts.

The supposed logic is that an LLM multiplies your skill. If the multiplier is 5, and your attacking skill is 1 before the multiplication, then you get 5 after; if your attacking skill is alreaady at 10, you get 50. You could argue that LLMs are not good enough to act as multipliers, and then my math won’t work.

[pron]

> A maintainer has a full-time job: to develop software. A maintainer who is also a defender has two full-time jobs,

I don't think so. This is already the situation. Maintainers already fix vulnerabilities when they know about them.

> On the other side there’s an attacker with a singular job and a strong incentive to do it well.

If the situation is that the attacker is focusing on a single project, the attacker will win, as they do already. But the attackers usually need to split their attention over lots of projects.

> The supposed logic is that an LLM multiplies your skill

I don't agree with that logic. Agents bring knowledge with them. That's not a multiplier. Compare how well a 12 year old can do compared to a Roman history professor on questions about Roman history when they both can use an LLM or when they both can't. The LLM will shrink the gap, not increase it.