[mk89]

Not as a metric, but it basically becomes one, like with Fedramp.

You need to fix also moderate/low CVEs within a certain time frame.

So CVE count becomes relevant, because the target is zero, although it doesn't mandate "zero CVEs" but that's finally what the desired outcome is.

It's basically unrealistic to ignore that number, because it's unlikely that you have a steady 1000 CVEs (that are being continuously fixed and new ones discovered), but more like "a few exceptions".

[tptacek]

I don't do FedRAMP and will have to take your word for that, but none of SOC2, 27001, or HIPAA/HITRUST care about CVE counts.

[kasey_junk]

PCI doesn’t mention cve by name but does require vulnerability accounting and requires action if they are found, the action required driven by severity. I could see a (poor) control being written around keeping counts down.

[tptacek]

Right, you can write a bad SOC2 control that cares about CVE counts too!

[tialaramex]

I've met good auditors. I mean, I've met terrible auditors too, but the good ones stick in my mind more because they ask insightful questions about my software or sometimes software in general. It's a problem that this is often seen as a box ticking exercise, done right it can be a really great opportunity to improve but so often instead the priority is to get the paperwork done and too bad if you achieved nothing by it.

[tptacek]

If we're talking about actual auditors, not tech consultants who call themselves auditors but people actually trained as auditors, I'd take it as a bad sign if they asked a bunch of specific unbidden questions about software details. That's not the job.

[tialaramex]

The specific example I'm thinking of most strongly is when we were purchased one of the auditors looking over our software noticed that unlike most of the big company's other software at that time (about 2012) ours was HTTPS-only. Their checklist told them they need only to check there's HTTPS for authentication pages, and they asked if that's actually enough, is it fine if everything else is just plain HTTP as they've seen elsewhere and as their checklist asks?

I of course said it isn't, because as we both know, it isn't.

"That's not the job" is I think the most useless possible observation here. The best outcome from audit isn't that you checked all the boxes, that's just resources expended for no benefit, the best outcome is that audit found a nasty problem early so that you could fix it now. The biggest problem we have in the Web PKI with auditors is that they'd so much rather tick boxes than tell their client - who they are billing $$$ - where the problems are. This presumably feels good to the suits, but if there's a problem and the auditors don't tell you the chances are somebody else finds it and then you're in worse trouble.