[HideousKojima]

About two years ago I was assigned to investigate several security tools, Veracode was one of them. For every tool I was able to get an actual demo/trial of, I tested them with a simple piece of code: 10 lines of C# that contained hardcoded credentials and a blatantly obvious SQL injection vulnerability. None of the tools I tried were able to detect both (though some would detect one or the other). IIRC (I tried a lot of different tools and wasn't able to get demos of all of them so it's hard to remember which did what) Veracode choked on any code using C# 8+ features like the simplified using blocks, which my 10 lines of vulnerable code used.

My recommendation to my manager and others was that for the price being asked for most of these tools, we could instead hire two or more devs to do nothing but code review and manually search for vulnerabilities and do a better job. Instead they went with Veracode, and the first thing I got got after we implemented it was three emails for false positives that didn't properly identify the code file and line number where the supposed vulnerabilities occurred, meaning I had to waste several hours communicating with the team that ran the scan to figure out where this supposedly occured.

I specifically remember two of the false positives were about my use of Serilog, saying my code was vulnerable to log injection. Even though Serilog isn't vulnerable to that if you use structured logging (which my code was). I even did several log injections and verified that they didn't actually work.

Edit: Looking at the reports it generated, I see it also failed to detect usage of the actually highly vulnerable BinaryFormatter class which I have seen used several times by our offshore teams. That's something that could be detected with a simple string search and they didn't detect it.

[MattPalmer1086]

No idea how good Veracode is in practice. It's been considered in at least 3 places I've worked and never been selected.

I would disagree that you can replace scanning tools with just human review though. You need both.

Static analysis and software composition analysis are good at flagging possible problems, but there's a lot of noise. You ideally want an application security engineer to be reviewing the code and scan results. And also conducting threat modelling with the devs, so you prevent problems even earlier.

[HideousKojima]

>I would disagree that you can replace scanning tools with just human review though. You need both.

If any of the scanning tools I've used were actually fit for purpose, I might actually agree with you, but they aren't. The amount of noise they generate makes them a distraction and a net negative.

[MattPalmer1086]

Actually, I have a question for you on this. There's a big push towards "shift left" in the industry.

I'm all in favour of empowering developers to deal with stuff earlier, but the amount of false positives these tools generate seem like it would just distract.

I think some kind of review and filtering of them before the average dev sees them would actually work better. Do you have an opinion on that?

[MattPalmer1086]

Completely agree there's a lot of false positives. On the other hand, having seen the number of actual vulnerabilities in code left after only manual review, it's something that's a necessary evil.

[neonsunset]

Unless using (very) old target (at which point is ouch), everything will scream at you for referencing anything related to BinaryFormatter without enabling extra analyzers. It might even be a build error at this point? (either way it's being removed .NET 9 after a lengthy deprecation grace period)