This isn't going to work because the LLM doesn't have enough context. Many security issues involve a failure mode which cuts across multiple parts of the code. A PR which seems perfectly valid on its own may be the missing piece which opens up a vulnerability. Each component may be fine on its own, but brought together, the system is vulnerable.
Think of a machine with interlocking gears; each gear may itself be perfect and may fit perfectly with each other, but then if a tiny pebble comes between them, the entire machine breaks. Maybe the problem here is that the final gear was too close to the ground and would catch stray pebbles kicked up by the wheel in front of it... The LLM couldn't know this unless it understood the full context in which the change occurred; not only the code, but the environment itself.
In a poorly designed codebase with hundreds of thousands of lines of code, it's impossible to have the full context of the code even. The architecture would lack proper separation of concerns to allow one to effectively establish an appropriate defense perimeter. In a poorly designed codebase, every part of the code can harbor a vulnerability.
It's like; if you don't have a proper access control layer which is automatically and declaratively enforced for all your endpoints, every endpoint will have to enforce security restrictions on their own; duplicating similar-looking code over and over. If one endpoint out of 1000 incorrectly enforces a security restriction, that could be a critical vulnerability.
Of course you don't just check the diff. Rather in your CI infra, it's important as part of every PR, it needs to be given the full repo to check if it introduces any issues. This works wonderfully on github, even with non SOTA tools like gemini-code-assist.
why do you think it's not possible to have full context of codebase? modern harnesses excel at finding all the right codepaths, even in a large codebase.
Think of a machine with interlocking gears; each gear may itself be perfect and may fit perfectly with each other, but then if a tiny pebble comes between them, the entire machine breaks. Maybe the problem here is that the final gear was too close to the ground and would catch stray pebbles kicked up by the wheel in front of it... The LLM couldn't know this unless it understood the full context in which the change occurred; not only the code, but the environment itself.
In a poorly designed codebase with hundreds of thousands of lines of code, it's impossible to have the full context of the code even. The architecture would lack proper separation of concerns to allow one to effectively establish an appropriate defense perimeter. In a poorly designed codebase, every part of the code can harbor a vulnerability.
It's like; if you don't have a proper access control layer which is automatically and declaratively enforced for all your endpoints, every endpoint will have to enforce security restrictions on their own; duplicating similar-looking code over and over. If one endpoint out of 1000 incorrectly enforces a security restriction, that could be a critical vulnerability.