[tptacek]

It's an audit standard about security. It's not a security standard. It defines a small number of extremely broad goals, like "you do risk management" and "you have access control mechanisms", which might be IT tools or might be a tabletop RPG.

You're irritated that people keep describing it at a security standard, which is understandable, but it isn't. AICPA auditors run SOC2 audits because SOC2 is an audit; it's about reconciling paperwork and evidence, about digesting policies and then checking that you actually do anything in those policies.

If you want to know about a firm's actual security program, you'll need to ask deeper questions than SOC2 can answer.

[alexjplant]

When I worked someplace undergoing a SOC2 audit I had to periodically jump into calls with our auditor and security architect to answer all sorts of highly-specific questions about how we deployed our software and the infrastructure that it ran on. At one point, for instance, the auditor told me that they needed me to demonstrate that our servers were all configured to synchronize their clocks to an NTP server. Kubernetes was a foreign concept to them and pointing to GKE docs wasn't sufficient - if memory serves I had to MacGyver some evidence together by hacking a worker node to be able to get a terminal on it and demonstrate that, yes, Google's managed VMs indeed run chronyd.

This seems to be the opposite of

> It's not a security standard. It defines a small number of extremely broad goals

Is this because of the specific auditors we were using? Are some more sympathetic than others to contemporary engineering practices?

[tptacek]

Yes, and yes. No matter how good your auditors are, unless you're accepting a shrink-wrapped set of controls from a tool provider like Vanta, you need to be pushing back on things they demand; you just have to have a clear idea of what the Common Criteria control they're looking for is (you'll see this clearly from the DRL they give you at the start of the engagement), and then when they ask for stuff that doesn't matter or isn't relevant for your org, you explain how what they're asking for has nothing to do with the actual control you're working on.

So far as I can tell there is almost nothing that is a firm requirement in a standard SOC2 Security TSC audit. We even got "background checks" rolled back.

Our audit firm is a SOC2 practice that informally spun of out of a Big 4 firm. When people get audits after using GRC tools like Drata, they often get matchmade to auditors who bid down the cost of the audit. It's possible that one of the things you get when you pay low-mid 5 figures for an audit instead of low-mid 4 figures for an audit is a lot more flexibility and back/forth with the auditors; I don't know. If that's the case: pay for the better auditors. These are rounding error expenses compared to doing extra engineering work just for SOC2.

[akerl_]

In my experience, it's more likely it was the approach of the folks at your company that made your controls.

SOC2 (and a bunch of similar regimes) basically boil down to "have you documented enough of your company's approach to things that would be damaging to business continuity, and can you demonstrate with evidence to auditors with low-to-medium technical expertise that you are doing what you've said you'd do". Some compliance regimes and some auditors care to differing degrees about whether you can demonstrate that what you've said you'd do is actually a viable and complete way to accomplish the goal you're addressing.

So the good path is that the compliance regime has some baseline expectation like "Audit logs exist for privileged access", and whoever at your company is writing the controls writes "All the logs get sent to our SIEM, and the SIEM tracks what time it received the logs, and the SIEM is only administered by the SIEM administration team" and makes a nice diagram and once a year they show somebody that logs make it to the SIEM.

One of the bad paths is that whoever is writing the controls writes "We have a custom set of k8s helm charts which coordinate using Raft consensus to capture and replicate log data". This gets you to the bad path where now you've got to prove to several non-technical people how all that works.

Another bad path is that whoever writes the control says "well shit, I guess technically if Jimbo on the IT team went nuts, he could push a malicious update to the SIEM and then log in and delete all the data", and so they invent some Rube Goldberg machine to make that not possible, making the infrastructure insanely more complex when they could have just said "Only the SIEM admins can admin the SIEM" and leaned on the fact that auditors expect management to make risk assessments.

The other bad path is that whoever is writing the controls doesn't realize they have agency in the matter, and so they just ask the auditors what the controls should be, and the auditors hand them some boilerplate about how all the servers in the server farm should run NTP and they should uninstall telnet and make sure that their LAMP stack is patched and whatever else, because the auditors are not generally highly technical. And the control author just runs with that and you end up with a control that was just "whatever junk the auditors have amalgamated from past audits" instead of being driven by your company's stack or needs.

[zdc1]

Similarly, I've had many instances where an auditor would ask for X and instead of trying to show them X I would instead ask them what control / Common Criteria item they were trying to get assurance on. So much of the process is about educating the auditors about how your systems operate and how you manage risks, rather than just trying to provide or build anything and everything they ask for.

*X = password expiry configuration, server antivirus, approval emails, etc.

[1dom]

> "whatever junk the auditors have amalgamated from past audits"

At a large financial company, I was tasked with gathering some audit data to evidence that only certain people could access certain things. To do that, we had to get the list of users with access.

The access control tool at the time used plain text files. I sent the plaintext file with the list of names to the auditor. The auditor said that won't do, because it could have been forged. That's fair.

After lots and back and fourths, the solution was that I needed to send over a screenshot of a terminal window with a list of names, because that's what the auditor expected, and that's what had previously been submitted.

Not a screenshot of the actual document. Not a terminal showing the hostname or similar on the server. I had to get the textfile I'd sent, open it in vim, take a screenshot of vim, and submit that.

[tptacek]

This is gold. The good-path bad-path thing is exactly the right way to think about it.

[close04]

Most of the bad paths are usually taken by engineers with little or no experience being audited. After going through the ringer a few times (learn not to answer questions that aren't asked, or that they have a say in what that control should be) the pendulum swings in the other direction, where the answers are always good-path, not necessarily the real-path. At least until the practical part of the audit starts to look at what they really do, not what they say they do.

There's another giant pothole to navigate in many organizations, related to this:

> when they could have just said (...) and leaned on the fact that auditors expect management to make risk assessments

When management has decision paralysis and fear of accountability the engineers feel the need to compensate for the tight spot and solve problems the way they know how to solve them. With technical measures. And a technical measure that fixes the organizational problem tends to be complex and fidgety. Doubly hard for the auditors to properly take in.

[akerl_]

“Management” here is a term of art. For many compliance regimes and controls, the engineer responsible for a system can make a statement as “management”.

[quicklime]

> Kubernetes was a foreign concept to them and pointing to GKE docs wasn't sufficient

This doesn’t surprise me one bit, in my case our auditors didn’t have a clue what GitHub was and we had to explain how code reviews and deployment pipelines worked. And these are the people who are tasked with certifying whether we’re doing our job correctly.

Sure, maybe it’s because we didn’t pick good auditors. But the accountants certified those auditors, and the whole point of certification is that we can rely on it to establish basic knowledge.

[tptacek]

You're relying on their ability to review documents and the meaningfulness of the reputation they stake on a signature saying they actually reviewed those documents. Nobody who has been through a SOC2 audit would ever reasonably think you're relying on your auditor's technology skills.

[er4hn]

I've always viewed SOC-2 as a certification for business continuity, not security. Once you view it as making sure that the service can continue running, even with disaster or heavy turnover, it makes more sense.