[tptacek]

If we're talking about actual auditors, not tech consultants who call themselves auditors but people actually trained as auditors, I'd take it as a bad sign if they asked a bunch of specific unbidden questions about software details. That's not the job.

[tialaramex]

The specific example I'm thinking of most strongly is when we were purchased one of the auditors looking over our software noticed that unlike most of the big company's other software at that time (about 2012) ours was HTTPS-only. Their checklist told them they need only to check there's HTTPS for authentication pages, and they asked if that's actually enough, is it fine if everything else is just plain HTTP as they've seen elsewhere and as their checklist asks?

I of course said it isn't, because as we both know, it isn't.

"That's not the job" is I think the most useless possible observation here. The best outcome from audit isn't that you checked all the boxes, that's just resources expended for no benefit, the best outcome is that audit found a nasty problem early so that you could fix it now. The biggest problem we have in the Web PKI with auditors is that they'd so much rather tick boxes than tell their client - who they are billing $$$ - where the problems are. This presumably feels good to the suits, but if there's a problem and the auditors don't tell you the chances are somebody else finds it and then you're in worse trouble.

[tptacek]

No, I mean, it's literally not the job. An auditor reconciles controls against reality. A "security auditor" parachutes into an environment and tries to flag as many security issues and gaps as they can find. They're very different jobs.