[simonw]

"The values passed to _sort were concatenated directly into SQL ORDER BY clauses with no validation" - sounds to me like this project had some low-hanging fruit!

Looks like every single one of the 38 vulnerabilities were either SQL injection, XSS, path traversal or "Insecure Direct Object Reference" aka failing to check the caller was allowed to access the record.

This is actually a pretty good example of the value of AI security scanners - even really strong development teams still occasionally let bugs like this slip through, having an AI scanner that can spot them feels worthwhile to me.

[tmoertel]

> Looks like every single one of the 38 vulnerabilities were either SQL injection, XSS, path traversal or "Insecure Direct Object Reference" aka failing to check the caller was allowed to access the record.

Seems like code review against a checklist of the most common vulnerabilities would have prevented these problems. So I guess there are two takeaways here:

First, AI scanners are useful for catching security problems your team has overlooked.

Second, maintaining a checklist of the most-common vulnerabilities and using it during code review is likely to not only prevent most of the problems that AI is likely to catch, but also show your development team many of their security blind spots at review time and teach them how to light those areas. That is, the team learns how to avoid creating those security mistakes in the first place.

[Falell]

I think it shows exactly the opposite of the second. Even with the availability of checklists, and instructions to use them, people won't and don't actually use them consistently.

'With enough eyes, all bugs are shallow' and AI is an automatable eye that looks at things we can tell nobody has seriously looked at before. It's not a panacea, there will be lots of false positives, but there's value there that we clearly aren't getting by 'just telling humans to use the tools available'.

See also: modern practices and sanitizers and tools and test frameworks to avoid writing memory errors in C, and the reality that we keep writing memory errors in C.

[Avamander]

> See also: modern practices and sanitizers and tools and test frameworks to avoid writing memory errors in C, and the reality that we keep writing memory errors in C.

I think there's a difference in how trivial some of these things are to detect and how difficult others are. IDOR and SQLi aren't nearly as complex as C unsafety is.

[capiki]

What about having the checklist and having an AI tool use it to catch things at review time (or even development time)?

[tmoertel]

Having AI tools do the review against the checklist would probably prevent the problems. However, it would probably be substantially inferior as a teaching tool for your team. The exercise of having reviewers hunt the checklisted vulnerabilities for themselves is what develops the mental muscles needed to understand the vulnerabilities in depth and avoid them when designing and writing future code.

But, yes, I'd augment any manual review with a checklist and AI review as a final step. If the AI catches any problems then, your reviewers will be primed to think about why they overlooked them.

[dylan604]

> The exercise of having reviewers hunt the checklisted vulnerabilities for themselves is what develops the mental muscles needed to understand the vulnerabilities in depth and avoid them when designing and writing future code.

Could not agree any more strongly. These automagic tools are one thing in the hands of a dev that groks the basics like these examples. It would be one thing if new devs were actually reviewing the generated code to understand it, but so much is just vibe coded and deployed as soon as it "works". I get flack from not immediately deploying generated code because I want to take time to understand how it works. It's really grating and a lot of friction is coming from it.

[capitalhilbilly]

For vulnerabilities of this nature is there really a point in training if an AI will catch them from now on? Seems like a variant of the allowing calculators problem and maybe the problem codeless platforms would have had. If these style of bugs don't change design in any meaningful way then the user can just write pseudo variables and the AI can normalize to safe code and their ability to work without the AI and IDE is probably less relevant than freeing their cognitive load for more complex constraint problems.

[ndriscoll]

Suppose we still need humans to be writing code and caring about this stuff for the foreseeable future, so we need people to continue learning about the ways things can go wrong. For something like injection, you still ideally have a lint rule that says "don't concatenate things that look like SQL/HTML/etc. Use the correct macros for string interpolation". What does it actually teach for a reviewer to tell you that? You can ask the reviewer for more information, but you can ask your teammate anyway if you don't understand why the linter is mad. You can also ask the robot, who will patiently explain it to you even long after all of the knowledgeable humans have retired or died. The robot could even link to a prompt asking to explain it:

https://chatgpt.com/share/69f10515-8808-83ea-abe3-a758d3144c...

If people aren't learning more with AI, that's a meta skill they need to develop.

As for training the review muscles, why would you do that if you have a linter that rejects when you make the mistake? I don't expect reviewers to check whether you eschew nulls or uninitialized variables; I expect the compiler to do that, and I expect over time that more and more things will become tooling concerns (especially given that rigid tools with appropriate feedback are clearly a massive force multiplier for LLMs).

[tmoertel]

Two issues here. First, teams that decide to delegate security responsibilities to AI are more likely to do things fast and loose, in general, and thus be less likely to "ask the robot to patiently explain" problems until they understand the problems' root causes and update their mental models to prevent those problems.

Second, to use your example, the ChatGPT response you provided does a crappy job of explaining the root cause of problem: Namely, that every string is drawn from some underlying language that gives the string its meaning, and therefore when strings of different languages are combined, the result can cause a string drawn from one language to be interepreted as if it were drawn from another and, consequently, be given an unintended meaning.

So, if the idea is that smart teams can not only delegate the catching of problems but also the explanation of those problems to ChatGPT -- presumably because it is a better teacher than the senior engineers who actually understand the salient concepts -- I'd say AI ain't there yet.

[ndriscoll]

> teams that decide to delegate security responsibilities to AI are more likely to do things fast and loose

Is that true? Is that also true of e.g. teams using type checkers to avoid nulls or exceptions? Or teams that use memory safe languages to avoid memory corruption? Or using a library that has an `unsafeStringToSql` API surface, and a linter to flag its use (where you're expected to use safe macros instead)? My experience is that better tools (or languages and library designs) scanning for issues lead to fewer defects and less playing fast and loose since the entire point of the tools is to ban these mistakes.

On education, it literally tells you that the top concern is SQL injection made possible by concatenating strings, and gives an example of an auth bypass: `name = "foo' OR 1=1 --"`. It also notes that this is not just a minor nitpick, but that actually the solution is fundamentally doing something completely different (query objects with bound parameters). If you don't understand what it means you can just ask:

> Elaborate on 1

> Walk through examples of what goes wrong and why, and how the solution avoids it

etc. The knowledge is all there; you just need to ask for it. It's an infinitely patient teacher with infinite available attention to give to you. You can keep asking follow-ups, ask it to check your understanding, etc. Or there are tons of materials about it on the web or in textbooks, and if you still don't understand, you can still ask a more senior engineer to explain what's wrong.

[ulimn]

Checking for OWASP top 10 items during code review is usually a mid level dev interview question IME. It's nothing new. Teams don't have to come up with these. These things exist.

[simonw]

Yee, absolutely. A team with a strong code review culture that incorporates security review against common exploits ideally wouldn't end up with holes like this.

[ihaveajob]

I guess the value of the tool is that it gives you that same benefit for the cost of a few tokens.

[tmoertel]

> I guess the value of the tool is that it gives you that same benefit for the cost of a few tokens.

But it doesn't give you the same benefit. It gives you the partial benefit of catching these problems before they go to production, but it doesn't give you the remaining benefit of teaching your team about where their mental models are broken. A team that decides to delegate this responsibility entirely to AI is going to have a hard time learning about these serious defects in their mental models. Fixing those defects will pay dividends throughout the code base, not just in the places where AI would detect security failing.

[ihaveajob]

Not if you treat it as a magical box that fixes things on its own. We're a tiny team and our process has improved a lot thanks to processing AI reviews and learning common patterns. It gets tiring to get the same feedback over and over so humans learn.

[dgb23]

But by not having a checklist you avoid that your blind spots get exposed.

[tmoertel]

Why would you want to prevent your development team from learning about their blind spots?

[dgb23]

I tried to make a joke about the tensions of security and accountability.

[dylan604]

So you can move faster to the next features obviously. Refactoring for secure code is time consuming, and clearly wasted cycles as the code is working. /s

[someguynamedq]

"should" and yet didn't

[camdenreslink]

I don't think strong development teams are still letting SQL injection vulnerabilities through by manually concatenating strings to build queries with user-provided data. Not in the year 2026.

[voxic11]

Keep in mind this project is a 25 year old PHP application.

[zarzavat]

That actually makes it more confusing since a 25 year old PHP application is exactly where you'd expect to find SQL injection vulnerabilities.

If I were in charge of a 25 year old PHP application, tracking down every SQL query and converting it to a safe form would high on my list of priorities. You don't need AI for that, just ripgrep and a basic amount of care for your users.

[whythismatters]

Most (proprietary) 25 year old PHP codebases I've seen are a huge mess riddled with issues, exuberant loc, mix of tabs and spaces and weird indentation, dry violations, slightly diverging code blocks copy-pasted all over the place, etc., etc. Resolving technical debt (let alone reviewing the "stuff that works" like SQL queries) is often low priority because it's tedious and does not create any "business value".

[otabdeveloper4]

Replacing/automating manual ripgrep is a top-1 use case for AI though.

[pseudalopex]

Their point was a competent team would have done this since 10 or 20 years I thought.

[simonw]

Good frameworks can protect against SQL injection and XSS (through default escaping of output variables) but protecting against insecure direct object access is a lot harder.

[tdeck]

Last time I had to build an ORDER BY clause in MySQL, it didn't support query parameters in prepared statements, which is probably how this happens. It's not an excuse at all but the standard path of "just throw a ? (or whatever) in there and use bound params" doesn't work for order by (or at least it didn't at some time in the recent past). You would end up having to concatenate strings somehow or other.

[IshKebab]

Yeah this is a huge red flag that would make me avoid this project for sure.

Unfortunately you have no easy way of checking if closed source projects are similarly amateur.

[Taters91]

These kind of checks were available without AI.

[sheikhnbake]

Math is doable without a calculator

[happytoexplain]

The headline is "AI uncovers...", implying that the standard static analyzers used by basically everybody didn't catch them.

[serf]

isn't this just sort of turning chicken-or-egg?

if an AI uses static analyzers to do work ,is it the tool or the ai ?

if AI is using grep to do the work, is it the AI or grep?

I mean essentially all agent work boils down to "cat or grep?"

[RA_Fisher]

AI gives us a means of leverage. We can do more with less. production = f(labor, capital, technology) + eps

[krainboltgreene]

This always comes up and the only thing I can think is: Doesn't Google make like 10B a quarter in profit from GCP alone? Did we really need a cheaper SQL injection checker?

[positron26]

Was the human labor?

[nudpiedo]

There are Static code analyzers which already would have detected that.

And these were also automatic. Looks very likely that the team didn’t give a damn about top basic security and good practices.

Like a house made of paper wouldn’t be an example of the insecurity of the construction industry.

[simonw]

Which static code analyzers do you recommend?

[happytoexplain]

SonarQube is extremely common, but I'm sure there are many.

[gchamonlive]

Isn't this something SonarQube catches?

[webXL]

Yes. Isn't this something code review catches? :)

[happytoexplain]

Sometimes, but not nearly as reliably as a static analyzer. But I'm assuming the unstated point you are sarcastically implying is "you don't need SonarQube" - maybe you're trying to say something else.

[gchamonlive]

WebXL makes no sense, because that's comparing oranges to apples.

[positron26]

Presuming there is an infinite pool of programmers who tirelessly work for a low price?

[gchamonlive]

I'm really curious what's your line of thinking here. Could you elaborate?

[gowld]

I think SQL Injection detectors were pretty mature even before the "AI" version?

[Groxx]

>even really strong development teams still occasionally let bugs like this slip through

agreed, though I think you'd be hard-pressed to find anyone who uses healthcare-related software professionally who thinks any "really strong development team" was involved in its creation.

[EGreg]

“even really strong development teams”

One would think a single really strong developer, let alone a team, would look for interpolation in strings fed to RDBMS?

[srveale]

And yet here we are

[prerok]

Everybody knew somebody should do it, but nobody did it.

Classic.

[gostsamo]

> This is actually a pretty good example of the value of AI security scanners

Are you fuckin' serious? This would be caught with any self-respecting scanner even 5 years ago and with most educated juniors even earlier.

I use AI every day, but I'm not deep enough in the dilulu to believe that everything above two brain cells should be a transformer.

[simonw]

Which scanners catch insecure direct object access?

[hilariously]

Honestly those all sound like common linters could find things like string concatenation.