[kasey_junk]

PCI doesn’t mention cve by name but does require vulnerability accounting and requires action if they are found, the action required driven by severity. I could see a (poor) control being written around keeping counts down.

[tptacek]

Right, you can write a bad SOC2 control that cares about CVE counts too!

[tialaramex]

I've met good auditors. I mean, I've met terrible auditors too, but the good ones stick in my mind more because they ask insightful questions about my software or sometimes software in general. It's a problem that this is often seen as a box ticking exercise, done right it can be a really great opportunity to improve but so often instead the priority is to get the paperwork done and too bad if you achieved nothing by it.

[tptacek]

If we're talking about actual auditors, not tech consultants who call themselves auditors but people actually trained as auditors, I'd take it as a bad sign if they asked a bunch of specific unbidden questions about software details. That's not the job.

[tialaramex]

The specific example I'm thinking of most strongly is when we were purchased one of the auditors looking over our software noticed that unlike most of the big company's other software at that time (about 2012) ours was HTTPS-only. Their checklist told them they need only to check there's HTTPS for authentication pages, and they asked if that's actually enough, is it fine if everything else is just plain HTTP as they've seen elsewhere and as their checklist asks?

I of course said it isn't, because as we both know, it isn't.

"That's not the job" is I think the most useless possible observation here. The best outcome from audit isn't that you checked all the boxes, that's just resources expended for no benefit, the best outcome is that audit found a nasty problem early so that you could fix it now. The biggest problem we have in the Web PKI with auditors is that they'd so much rather tick boxes than tell their client - who they are billing $$$ - where the problems are. This presumably feels good to the suits, but if there's a problem and the auditors don't tell you the chances are somebody else finds it and then you're in worse trouble.

[tptacek]

No, I mean, it's literally not the job. An auditor reconciles controls against reality. A "security auditor" parachutes into an environment and tries to flag as many security issues and gaps as they can find. They're very different jobs.