[Ukv]

A summary, to my understanding:

* Their software reads config files to determine which behavior to monitor/block

* A "problematic" config file made it through automatic validation checks "due to a bug in the Content Validator"

* Further testing of the file was skipped because of "trust in the checks performed in the Content Validator" and successful tests of previous versions

* The config file causes their software to perform an out-of-bounds memory read, which it does not handle gracefully

[Narretz]

* Further testing of the file was skipped because of "trust in the checks performed in the Content Validator" and successful tests of previous versions

that's crazy. How costly can it be to test the file fully in a CI job? I fail to see how this wasn't implemented already.

[denton-scratch]

> How costly can it be to test the file fully in a CI job?

It didn't need a CI job. It just needed one person to actually boot and run a Windows instance with the Crowdstrike software installed: a smoke test.

TFA is mostly an irrelevent discourse on the product architecture, stuffed with proprietary Crowdstrike jargon, with about a couple of paragraphs dedicated to the actual problem; and they don't mention the non-existence of a smoke test.

To me, TFA is not a signal that Crowdstrike has a plan to remediate the problem, yet.

[hrpnk]

They mentioned they do dogfooding. Wonder why it did not work for this update.

[xh-dude]

They discuss dogfooding “Sensor Content”, which isn’t “Rapid Response Content”.

Overall the way this is written up suggests some cultural problems.

[stefan_]

You just got tricked by this dishonest article. The whole section that mentions dogfooding is only about actual updates to the kernel driver. This was not a kernel driver update, the entire section is irrelevant.

This was a "content file", and the first time it was interpreted by the kernel driver was when it was pushed to customer production systems worldwide. There was no testing of any sort.

[broknbottle]

All these people claiming they didn’t have canaries. They actually did but people are in denial that they are the canary for crowdstrike lol

[cjbprime]

It's worse than that -- if your strategy actually was to use the customer fleet as QA and monitoring, then it probably wouldn't take you an hour and a half to notice that the fleet was exploding and withdraw the update, as it did here. There was simply no QA anywhere.

[modestygrime]

Just reeks of incompetence. Do they not have e2e smoketests of this stuff?