[jmuguy]

Maybe the IT departments at the affected orgs take solace in the fact that so many other orgs had issues that the heat is off - but in my opinion this was still a failure of IT itself. There's no reason that update should have been pushed automatically to the entire fleet. If Crowdstrike's software doesn't give you a way to rollout updates on a portion of your network before the entire fleet, it shouldn't be used.

[candiddevmike]

The update bypassed the controls orgs had in place to defer/schedule updates, AFAIK.

[jmuguy]

I've had trouble nailing down if thats the case from searching around online. And if thats true - thats absolutely on Crowdstrike. And that behavior should disqualify it from being used on critical systems. I imagine this incident will cause a lot of teams to consider just what can happen automatically on their systems.

[wisemang]

It’s definitely the case. See Crowdstrike’s preliminary post incident review here: https://www.crowdstrike.com/falcon-content-update-remediatio...

The nature of “content updates” vs a full product update. Though you may be right, perhaps they provide controls for those updates, I’ve never used their software. But doesn’t sound like it.

[CydeWeys]

It's on CrowdStrike, but it's also on IT for even allowing installation of critical software like this that has a bypass at all. Updates shouldn't even be allowed to bypass IT's safe rollout procedures, at least not without IT signing off on it anyway.

[kelnos]

If that's the case, that doesn't change GP's point: if Crowdstrike can bypass your org's controls on rolling out updates to its software, it shouldn't be used.

[b4ckup]

Didn't day say in their incident report that they have a batched rollout strategy for software updates but this was a config update and the update path for configs does not have such a mechanism in place.

[bonestamp2]

Ya, so hopefully it's obvious to them that every rollout needs some kind of batching. I get that all devices within one org might need to have the same config, but in that case batch it out to different orgs over 2-3 days.

Maybe the more critical infrastructure and health care orgs are at the end of that rollout plan so they are at lower risk. It's not ideal if one sandwich shop in Idaho can't run their reports that day, but that's far better than shutting down the hospital next door. CrowdStrike could even compensate those one system shops that are on the front line when something goes down.

Again, better to pay a sandwich shop a few thousand dollars for their lost day of sales than get sued by the people in the hospital who couldn't get their meds, x-rays, etc in time.

[qaq]

Generally none gates content updates as they happen multiple times a day

[lucianbr]

Management decides to use Crowdstrike, not IT, and IT has no way to rollout updates in controlled fashion.

So not really a failure of IT, at least not for this reason.

[hello_moto]

In big companies, it's the Management of the IT team.

I know, not really the DailyWTF materials that majority HNers led to believe.

[jmuguy]

My comment assumes that the IT department (including its executive) gets to make these sort decisions - why wouldn't they?

[mingus88]

In many mature orgs, corporate IT rolls up to the CIO and security will roll up to the CISO

The CISO and security ops will demand to be completely independent from corp IT, for legit reasons, as the security team needs to treat IT as potential insider threat actors with elevated privileges.

They will also demand the ability to push out updates everywhere at any time in response to real-time threats, and per the previous point they will not coordinate or even announce these changes with IT.

There has always been an implicit conflict between security and usability, because of the inherent nature of security deny policies, but they also inherently conflict with conservative change management policies such as IT slow rolling changes through lower environments on fixed schedules and operating with transparency

[valenterry]

> The CISO and security ops will demand to be completely independent from corp IT, for legit reasons, as the security team needs to treat IT as potential insider threat actors with elevated privileges.

I always wondered: why should security ops not be a potential insider thread actor? In fact, if they were compromised, it would be even worse.

Do we need two different security ops that monitor each other? :)

[briffle]

In most clustered systems, you need at least 3 observers, so that its a clear majority of systems can decide that the observer is not working as expected.

So I guess 5 security OPS teams in different regions of the world, and they can all call a vote if one of the teams is now 'bad' :)

[com]

Generally, act vs monitor is the segregation of duties that I have seen best working between platform or IT ops and engineering (act) vs security ops (monitor).

For many high privilege operations there are more segregation of duties in the act side of things - these can be down to plan, authorise, configure, activate, validate or some rollups of these. Another is dual control on the act side, since conspiracy is generally quite hard to do especially if it’s just for pocket-change. Different if it’s $$Billions of fungible cash of course at stake.

People often overcomplicate - simple do/check is often enough.

[jachee]

Isn’t that why some organizations have a red team and a blue team?

[com]

Security should not, in general, have anything but awooga-awooga red lights and sirens break-glass write/change/delete/shutdown access to prod infrastructure or systems, or indeed anything that could compromise them. I’d argue that access to read or copy sensitive data is almost, but not quite as dangerous without extensive controls and internal monitoring too…

IMO there are no legit reasons except politics, empire building, NIH and toxic relationships for such a such a crazy state of affairs.

[Retric]

Major purchases tend to be pushed up the ladder. It’s not uncommon for a CEO or non technical director etc to decide what IT systems to use.

[fullspectrumdev]

In my experience the decisions on any non trivial IT system rollout are made by entirely unqualified, non technical execs who are usually swayed by marketing such as clownstrikes Super Bowl advertisement.

Technical people will make a recommendation, knowing it’s going to be ignored and that the decisions already been made.

[patmcc]

IT doesn't steer the ship in banks (and bank-like orgs). IT gets a mandate from the real decision makers that they have to choose something that does x, y, z - see "Regulations which strongly suggest particular software purchases" in the article for examples of x, y, z.

So sure, IT gets to "decide" - between CrowdStrike, SentinalOne, or Palo Alto (and maybe a couple others). But they don't really have much choice, they can't use an OSS solution, or roll their own, or anything else. They have to pick one of a small number of existing solutions.

[lucianbr]

Sure, if there's a security exec who made the decision, it may be their fault. I was thinking more of the rank and file, but that's just my bias.

[hello_moto]

When a ransomeware attack is happening, organizations will engage to cybersecurity vendors and will start PoC with a bunch of them and list the pros and cons of each vendors before they negotiate and ended up selecting the winner.