[taspeotis]

I don’t have much sympathy for CrowdStrike but deploying slowly seems mutually exclusive to protecting against emerging threats. They have to strike a balance.

[zavec]

Even a staged rollout over a few hours would have made a huge difference here. "Slow" in the context of a rollout can still be pretty fast.

[bboygravity]

But it can also still be way too slow in the context of an exploit that is being abused globally.

[taspeotis]

Sure but GP is praising "deploy so slowly that people complain."

[getcrunk]

Seriously like rolling out on some exponential scale even over the course of 10 minutes would have stopped this dead in its tracks

[yardstick]

In CrowdStrikes case, they could have rolled out to even 1 million endpoints first and done an automated sanity/wellness check before unleashing the content update on everyone.

In the past when I have designed update mechanisms I’ve included basic failsafes such as automated checking for a % failed updates over a sliding 24-hour window and stopping any more if there’s too many failures.

[goalieca]

They need a lab full of canaries.

[raffraffraff]

yeah, I don't get the "we couldn't have tested it" crap, because "something happens to the payload after we tested it". Create a fake downstream company and put a bunch of machines in it. That's your final test before releasing to the rest of the world.