[deepsun]

I'm still amazed how the blame shifted from Microsoft to CrowdStrike. Yes, CrowdStrike update caused that -- but applications fail all the time. It was Microsoft's oversight to put it on Windows critical path.

And banks/airlines etc were hit hard because their _Windows_ didn't boot, not because of an application crash on a perfectly working Windows.

[ctxc]

The application (Crowdstrike) was part of Windows' booting process.

Windows cannot simply "skip" failed drivers. Say Crowdstrike driver failed as a one time thing, Windows skipped it instead of retrying which led to the endpoint being vulnerable and a ransomware happens. We'd be saying the opposite now.

This is a high-impact ability Windows offers to applications - and applications should take responsibility and treat it as such.

I spoke to another EDR lead I know - they said they had provisions in place to read the dump if boot crashed, check if it was due to their driver and skip it if it was (and then send telemetry after startup so that it can be fixed, probably). Crowdstrike should have done the same.

One more thing to note is that we cannot say Windows shouldn't provide this ability - that becomes an anti-trust monopoly, because MS themselves are a competitor in this space.

[mewpmewp2]

But then again ransomware would happen like you said if they skipped it? And ransomware sounds even worse.

[burnished]

The difference is that if windows does the skipping then you probably don't find out until its too late, if the application does the skipping there is the opportunity to set up alerting so you can fix whatever went wrong.

[mewpmewp2]

Do you mean that the skip would be manually approved after telemetry is sent and folks on-call paged? Then that sounds like it could be viable and a good idea yes.

But always a chance that the skipping mechanism could break as well. And there must be some form of networking available to able to send that and ask for approval.

[ctxc]

Exactly! On skipping mechanism breaking - I mean, anything could break. Boils down to design and testing like all things.

One change - this approval and telemetry doesn't happen during the boot loading process. It's just logged and skipped.

Once bootup is done, the EDR app auto starts, checks logs for anomalies and sends telemetry over whenever network is available (it usually is, because they update malware signatures etc frequently). Someone at the company gets paged, they fix and the process continues.

[makeitdouble]

Windows could sure handle this kind of error better, but IMHO it would be a mistake to require Microsoft to absolutely block any path Windows could be crashing due to third party software.

We'd end in a situation similar to Mac OS where there's a single gatekeeper and whole industries are subjected to the will of the platform owner.

Enterprises have chosen Windows because of that flexibility and control, while having a business partner they don't get with linux. If anything the blame should fall on them for getting hosed even as they fully had the means to avoid that situation.

[CydeWeys]

I don't think "Microsoft should lock down Windows so hard" is the solution we want here. I don't want my desktop OS to be a walled garden like iOS is. I want to be able to install software on it that does anything I need to be able to do -- and yes, having that capability to run software at the lowest possible level in the OS does also mean that that software has extra responsibility to be well-behaved, as the OS can't protect the system from it. But I still would rather have that option than not have it (and also I wouldn't use CrowdStrike).

[klodolph]

How did Microsoft put it on the Windows critical path? (Informational question—I’m not following the issue super closely, but I thought CrowdStrike was a third-party system. Crowdstrike was wrong to put so much code in the kernel. Microsoft was reportedly legally bound to provide this access and allow third-party code to run in the kernel.)

[shombaboor]

There was an interesting article that these third parties who lobbied to run in the kernel and microsoft acquiesced about 20 years ago which led us down this path. https://web.archive.org/web/20061023112233/http://software.s...

[musjleman]

If you dig a little more about what this is talking about, Microsoft did not actually make any kernel related changes.

This was just Symantec and McAffee ranting about PatchGuard and MS did not remove it.

[dblohm7]

Microsoft added a feature to Windows that allows specially-signed antimalware drivers to be loaded extremely early in the boot sequence and be marked as non-optional. The idea is to give antimalware drivers the opportunity to load first, before anything else has had the chance to start.

Furthermore, if a driver is marked as optional and crashes, Windows can reboot with that optional driver disabled next time, preventing infinite crash/boot loops. Obviously that's no good if your antimalware driver gets disabled, so they can mark theirs as "required." Obviously in the CrowdStrike case, we got the worst of both worlds.

[umanwizard]

Microsoft is not who made the decision to put this on Windows' critical path; CrowdStrike was. Nothing stops you from running whatever dodgy third-party kernel modules you like on Linux or FreeBSD and they could easily cause the same sort of problem.

[Bjartr]

In fact, CrowdStrike has taken down Linux systems in much the same way in the past year (in April I think). It's just that the impact was less widespread.

[deepsun]

Linux yes, but *BSD systems have microkernel architecture, so must be more resilient to failures of one of the components. Although I have no idea whether the full system would boot either, I'm pretty sure it could partially load, give more information to user, and make it easier to fix.

[deepsun]

Partially agree. Linux yes, but *BSD systems have microkernel architecture, so must be more resilient to failures of one of the components. Although I have no idea whether the full system would boot either, I'm pretty sure it could partially load, give more information to user, and make it easier to fix.

[imiric]

To be fair, AFAIK the CrowdStrike driver was WHQL-certified. The loophole is that the driver loaded files at runtime, which made it impossible to predict every failure scenario.

Maybe this is the loophole that needs closing. You can't claim a driver is certified for Windows if the manufacturer can push arbitrary files that change its behavior. Especially if that manufacturer has sloppy development practices.

I understand that a primary goal of endpoint monitoring software is to be able to quickly react to new threats, and that the turn around time for Windows certification is surely unacceptable in this scenario, but this functionality can never be allowed to jeopardize the stability of the system it's supposed to protect. So it's ultimately on Microsoft to fix this for their users.

[shadowgovt]

Ironically, this is exactly the failure pattern that the changes in Chrome extensions to manifest v3 try to prevent. You can't provide a guarantee to the end-user of pre-vetted safety when the application is downloading and executing arbitrary code from a third-party source. That's like expecting a static code verifier to prevent all runtime errors.

It is, perhaps, a guarantee that no vendor should be expected to make.

[SoftTalker]

> You can't provide a guarantee to the end-user of pre-vetted safety when the application is downloading and executing arbitrary code from a third-party source.

So a web browser can't be trusted or certified, ever. Unless JavaScript is disabled?

[blackoil]

JS lives in a sandbox, that will require a bug to escape. Plugins are out of sandbox and random plugins should be disabled if security is a concern.

[shadowgovt]

Correct, and I should have been more clear. By the nature of what they do, Chrome extensions operate outside the sandbox designed to make attacking the underlying operating system running the browser very hard.

Sandboxing is such a way to attempt to enforce a guarantee (modulo sandbox bugs, of course). Since crexs aren't entirely in the sandbox, vetting and signoff is supposed to provide the added assurance of security the sandbox can't provide. And those assurances are hollow when the vetted crex is running arbitrary code from a third-party source.

[Cthulhu_]

In the article it states that Microsoft HAD to allow Crowdstrike to run in kernelspace by EU laws, because else MS would have the monopoly on kernel-level security solutions / integrations.

[Macha]

They probably had to, in the same way that banks had to use crowdstrike. Much as it's easy for banks to say "we use crowdstrike, like everyone else" rather than implement a bespoke and accountable framework for risk assessment and mitigation for every type of endpoint use case (and argue that case to both the auditor and regular). In this case it's easier for Microsoft to say "see, they can run in kernel space" rather than provide a bunch of API functions that achieve what's needed, convince all third party vendors to use them, and put in place a process to convince an auditor that Microsoft security software will never use any knowledge or functionality from the OS outside this.

[ghusto]

Exactly this. Microsoft did this poorly, so they were forced to allow others to do things poorly too.

[Macha]

I guess I don't think that's the sole reason, as I think the incentives would still be in place even if Microsoft authored security software did not run anything in kernel space.

[ghusto]

You mean in terms of third-parties wanting that level of access regardless? I agree, but it would be an easy "no" then.

[Macha]

In terms of Microsoft convincing regulators that they aren't and won't use any OS knowledge or private APIs ever.

[fmbb]

Did they have to?

Or did they choose to keep their own security software to run in kernel space thus forcing themselves to let others play by the same rules?

[marcosdumay]

They had to allow the same kind of access they have on their own "security" software.

Nothing in that means they need ring-0 access.

[davidgerard]

So why didn't MS lock it down in the US if it's an EU-local rule? Their excuse isn't plausible.

[voytec]

You're spilling cheap propaganda. Microsoft likely never had[0] an appropriate userland-level API in place and them blaming the EU should not be repeated by someone calling themselves a journalist.

[0] https://www.youtube.com/watch?v=EGttFWntctU - I need to state here that I do not possess the level of knowledge the author of video presents and therefore am unable to confirm findings included in the video

[deepsun]

> Microsoft likely never had

And we're back to Microsoft -- they are responsible for not having a proper way to handle such third-party apps, nor they maintained a process and controls to prevent such rogue breaking updates.

[asr]

Not MSFT’s fault: https://stratechery.com/2024/crashes-and-competition/

[999900000999]

I’ve used this analogy before.

If I sell you a bike and you remove the breaks you can’t sue me when you crash.

Any OS which allows users to do what they generally want to do, also allows users to fubar their own systems.

[deepsun]

Let me exaggerate a bit to show how bad that analogy is:

Let's say I've developed an laptop that bricks whenever you open a website with incorrectly formatted HTML.

Not sure how to adapt your bike analogy to this... Let's say you made a bike that's intended to be ridden outdoors, but breaks down whenever user sits on indoors. Yea, no one is supposed to ride it indoors. Not sure it's the best analogy though.

UPDATE: let's say the bike breaks down completely whenever it's ridden in the rain.

[999900000999]

No one's forcing you to install kernel level software.

If I install some kernel level anti cheats and they stop Windows from booting, I need to blame the game developers. Not Microsoft.

Your free to install pretty much whatever you want on Windows.

[Saris]

What about the previous crowdstrike bugs that hit Linux systems in a similar fashion?

I don't understand how this has anything to do with Windows, Crowdstrike is the one who built the application.

[deepsun]

It has everything to do with Windows, because it's Windows who crashed.

Applications crash all the time. But in this case people weren't able to even load the Windows to figure what's wrong or what app has crashed.

Microsoft allowed a third-party to self-update and didn't put a proper system of review and updates control to the heart of its OS.

[Saris]

The same thing happened before with Linux, crowdstrike made systems unbootable.

So I don't understand why you're focusing on windows here. Linux allows anyone to update too, there's no review or control either.

Just because an OS allows you to break it, does not mean the maker of the OS is liable when you do break it.

[deepsun]

Same with Linux yes, I never said Linux is any better in this question than Windows. At least it's free, and no warranties is given. But if RedHat had failed the same way, I think ReHat Inc would bear the blame just as well.

PS: I believe BSD-based systems would be more resilient because of microkernel architecture.

[btbuildem]

Isn't corporate malware by definition on the "critical path"? The article outlines the reasons why that jank runs in kernel space, and why MS is unable to "downgrade" it to userspace.

[ClumsyPilot]

This is the comment I expected, begging to handover your freedoms to run software to a big carry.

If you replace parts in your BMW, and put in some garbage or incompatible parts, it your fault if it doesn’t run.

You expect to sue your mechanic if he messed up, and for him to cover the full cost. For some reason people do not expect CrowdStrike to pay for their stupidity, which is the root of the problem. And the management that installed crowdstrike without due diligence

[ClumsyPilot]

This is the comment I expected, begging to handover your freedoms to run software to a big carry.

If you replace parts in your BMW, and put in some garbage or incompatible parts, it your fault if it doesn’t run.

You expect to sue your mechanic if he messed up, and for him to cover the full cost. For some reason people do not expect CrowdStrike to pay for their stupidity, which is the root of the problem. And the management that installed crowdstrike without due diligence

[deepsun]

Bit it wasn't some garbage parts in a car, it was an app. And apps fail all the time, OS is expected to handle that. Same as car is expected to handle rain for example.

[vel0city]

Buggy third-party kernel modules cause kernel panics all the time in Linux. You can easily write a kernel module to make a Linux system explode.

[ClumsyPilot]

> it was an app. And apps fail all the time

Exactly

The fact that developers do not take their responsibility as seriously as an average car mechanic bring shame on our entire industry

[ortusdux]

Is there any merit to Microsoft's argument that the EU forced them into keeping their kernel accessible by 3rd parties?

https://www.theregister.com/2024/07/22/windows_crowdstrike_k...

[kmeisthax]

The EU's rules are that Microsoft can't hoard APIs away from competitors, not that they have to give competitors a kernel driver SDK. If Microsoft says Windows Defender needs a kernel driver, then CrowdStrike gets to ship a kernel driver, too.

Microsoft, interestingly enough, is working on a project to add an eBPF[0] runtime to the NT kernel. If they were to use this for their own security products then I doubt the EU would prohibit them from transitioning third-party security products to eBPF programs. Antitrust and competition law do not care about specific technical measures competitors use to compete, just that dominant companies are not shutting competitors out of markets.

[0] Formerly "extended Berkley Packet Filter", eBPF lets you run safety-verified code in kernel space. Notably, the verifier isn't just a signing check, it can actually ensure the code won't crash the kernel directly.

[ghusto]

Yes and no. As others have pointed out above, it is factually correct that they were forced by the EU to give access to kernelspace. However, it is also true that the only reason for that was that _they_ were using kernelspace for the same things (instead of creating a framework and API into the features needed).

[davidgerard]

No. They could have done an EU-only edition that behaved that way. But they didn't.

[gciguy]

Dave's Garage has a great video on this: https://www.youtube.com/watch?v=wAzEJxOo1ts

[kmeisthax]

Microsoft didn't write the Falcon sensor software nor did they put it in the kernel. In fact, Microsoft has been shouting to the heavens trying to shift the blame from CrowdStrike onto the European Commission, because they want people to irrationally hate antitrust so they can turn Windows into shitty iOS and monopolize the security market (and applications market) for it.

Furthermore, Microsoft does actually have some rules regarding what you can and can't put into a signed kernel driver. Specifically, they won't sign kernel code unless they've seen and tested it first. CrowdStrike deliberately circumvented this rule by implementing their own configuration format - really, just a fancy way of loading code into the kernel that Microsoft doesn't have signing control over.

If there is blame to be had here for Microsoft, maybe it's that their kernel code signing program doesn't scrutinize third-party configuration formats hard enough. I mean, if you sign a code loader, you're really signing all possible programs, making code signing irrelevant. And configuration is more often than not, code in a trenchcoat. It's often Turing-complete, and almost certainly more complicated than the actual programming languages used to write the compiled code being signed off on.

But at the same time I imagine Microsoft tried this and got pushback. That might be why they feel (incorrectly) like they can blame the EU for this. Every third-party security solution does absolutely unspeakable things in kernel space that no one with actual computer science training would sign off on, using configuration to wrestle signing control away from Microsoft. Remember: Crowdstrike is designed to backdoor Windows systems so that their owners know if an attack has succeeded, not to make them more secure from attacks in the first place. Corporations are states[0], and states fundamentally suffer from poor legibility: they own and operate far too much stuff for a tribe[1] of humans to meaningfully control or remember.

The problem is that we have two different entities that all have the ability to stop this madness. When states run into this situation, they impose "joint and several liability", which means "I don't care how we precisely assign blame, I'm just going to say you all caused it and move on". In other words, it's Microsoft's fault and it's CrowdStrike's fault.

[0] ancaps fite me

[1] Maximally connected social graph with node degree below Dunbar's number.

[supriyo-biswas]

> because they want people to irrationally hate antitrust

One only needs to look at what's happening with Google's privacy sandbox to know the perils of antitrust with regard to introducing new interfaces. Even though Google has offered new interfaces and APIs that they themselves intend to migrate to (and take a ~20% revenue reduction), they've attracted the scrutiny of regulators who claim that this is a way of locking out competitors in the advertising space.

> [0] ancaps fite me

This part is simply inciting a flamewar, and something that you can do without in the spirit of the website guidelines[1].

[1] https://news.ycombinator.com/newsguidelines.html

[kmeisthax]

It's important to remember that every other browser dropped third-party cookie support years before Chrome did. Google dragged their feet on it until they could come up with a solution that would give Google the same level of tracking, because Google is an advertising company. So the competition authorities are telling Google - and only Google - that they can't drop third-party cookies anymore.

I've never actually heard anyone claim Privacy Sandbox[0] APIs would give third-party ad networks the same level of tracking as Google. But I imagine even if they did, the APIs would probably be a poor fit for competing ad networks, in the same way that, say, the iOS File Provider APIs are a terrible fit for Dropbox[1].

There are three different ways you can introduce a new standard or interface:

- You can go to or form a standards body with all the relevant market players and agree on a technical specification for that interface. This is preferred, and it's how the Web is usually done.

- You can take a competitor's interface people are already using and adopt that. This is how you get de-facto standards, and while they might have loads of technical problems[2], none of them give you an unfair market advantage.

- You can make your own interface and force competitors to adopt that. You get all the technical problems of a de-facto standard, but those are all problems your competition has to deal with, not you.

The difference is a matter of market advantage. Out of all the major browser vendors, only Google has dominance in online marketing. Microsoft and Apple would like to have a piece of that pie, but they all dropped third-party cookies without tying it to their own competing standards that they wanted to force other people to use.

[0] Hell of an Orwellian name

[1] For example, if you use Dropbox as your file storage, you can't pick folders. At all. On an operating system built by the company whose engineers are obsessed with bundles (directories that look and act like files instead of folders).

[2] laughs in SWF

[hulitu]

I think they said it was a windows driver, not a normal application. Running crap in kernel mode does not end well on any OS.

[concerned_user]

Yes it is a driver which is signed and tested by Microsoft. Driver allows to run arbitrary unsigned code. Why is that allowed?

[cyberpunk]

The driver is some kind of AV/Signature detection hook. E.g check every open() for this list of checksums and refuse to open known viruses style system. The 'update' was a borked definition file which triggered a bug in that system.

It's not code execution without signing, and I think probably they do want these files to be updated hands free.

The real problem was the lack of testing, rather than the actual mechanism I think.

[shadowgovt]

This is the nugget of the issue. The code-signing process, in this case, was abused to verify something that, fundamentally, cannot give the guarantee "Doesn't crash your OS" because it is allowed to run arbitrary code in the form of novel commands in what is essentially a DSL. So if code-signing is supposed to be a guarantee from MS that "this code can't crash your system," it should never have been signed... But then MS would have been on hooks for blocking a competitor.

There is no guarantee the law is written soundly.

[wolpoli]

To get a driver signed by Microsoft, the developer of the driver is required to provide a full cert pass log from the Windows Hardware Lab Kit to dev center [0]. Do you have any article that says the CrowdStrike driver has been tested by Microsoft?

[0]: https://learn.microsoft.com/en-us/windows-hardware/drivers/i...

[rtkwe]

To avoid going through the full cert process the sensor was certified but it loaded code from an uncertified module too so that it could be quickly updated to catch new threats. It's a tough corner to be in, to function properly it needs to update very quickly but the cert process takes a while to complete so they went with this work around of a signed module loading uncertified code.

[Joker_vD]

...you want Microsoft to forbid you from running certain kinds of programs on your own machine, even if you really, really insist on it, do I understand you correctly?

[hpen]

More like: "...you want Microsoft to forbid you from running certain kinds of programs (with gaping security holes / processes) on your own machine" YES

[Sohcahtoa82]

> (with gaping security holes / processes)

The problem is that you're assuming you can prove a program doesn't having security holes and bad processes.

[hpen]

You're moving the goal post waaaay far down. How about just following best practices? How about not allowing runtime code injection? Turns out security holes often have much in common, and with ways to mitigate them. Stop 100% of security holes? nah. Stop 99.9% of security holes? Yes and what an improvement.

[Sohcahtoa82]

The Crowdstrike failure was not caused by running unsigned code.

[hpen]

This is a valid opinion and I don't know why you were downvoted (well other than the hacker news bubble mindset (or mindless-set).

How is Microsoft not to blame, it's their product? We wouldn't blame a Toyota supplier for a failure in a car, but we somehow segment that in the software world?

[vel0city]

Toyota chose the supplier, worked with them on the specs and designs, and put it in their OE car delivered to the customer. It has Toyota's name on it, it was bought at a Toyota dealership, is a part of Toyota's warranty.

Crowdstrike is entirely optional software that doesn't come from Microsoft. Microsoft doesn't market it. Microsoft had no hand in making it. Microsoft doesn't sell it. Microsoft had no hand in a user installing Crowdstrike.

Do you not see the obvious differences there?

[Sohcahtoa82]

> How is Microsoft not to blame, it's their product?

Do you think Crowdstrike is a Microsoft product?

[hpen]

No. My point is that Microsoft allows the damn thing to be ran in kernel space. Mac, linux don't have this problem due to how THEY architected the system. Yes I think that puts Microsoft at blame.

[Bjartr]

> linux don't have this problem due to how THEY architected the system

This is incorrect. CrowdStrike caused a similar "won't boot after update was pushed" issue on Linux earlier this year, see https://news.ycombinator.com/item?id=41005936

[vel0city]

> Microsoft allows

Microsoft should have no say to decide what software I am allowed to run on my computer.

> Mac, linux don't have this problem due to how THEY architected the system.

You're joking right? You're arguing kernel panics can't happen on Linux? FFS, the CrowdStrike sensor caused kernel panics on multiple Linux distros in the last few months! Linux is not immune to kernel panics for buggy kernel modules.

[hpen]

The first point is pretty philosophical so I'm not gonna go far into that. At the end of the day you bought a product from a company, some of those products have a way to load programs on and some are locked down (a microwave). "should" is pretty biased whether I agree with that conclusion or not.

Two: Here I'm not arguing about what's possible but rather what happened in the real world. 8.5 M machines down, my org runs Macs, we knew about it from the news...

[vel0city]

You're arguing for a microwave that would only allow you to put in foods approved by the manufacturer in it.

And yes, in the real-world, third-party software can and does cause Macs to crash.

8.5M machines...out of what, 1.4 billion? That's what, 0.6% of machines?

[jsmith99]

Yes, use a different operating system, one that gracefully handles null pointer dereferencing by third party kernel modules? /s