[throwaway3306a]

Presumably they would fix these computers first thing during the night from a backup? If not, is this really about CrowdStrike, and not about a hospital unable to keep their absolutely critical computers backed up and restored in a timely manner?

Again, I understand that restoring a complex net of servers is hard and takes time. But they surely have local hospital IT admins for these absolutely critical computers who are always available on site and can do it individually - it's not like there will be more than a hundred of these at a particular hospital? Hack it a little if you have to, disable the SSO etc - all that can be fixed later.

[advael]

The unfortunate fact of the matter is that centralizing IT systems around large corporate products, including the on-prem software and any cloud services, necessarily means less local control of what can go wrong and how it can be mitigated, and thus often problems that simply can't be fixed, even by competent on-prem staff. Even when it is possible, it's often highly illegal, and most organizations do a lot to beat risk-aversion into everyone on their staff, and of course I mean aversion to risk of breaking rules or protocols, not risk like "someone dying"

I think it's always a mistake to outsource control of a mission-critical system, but that is exactly what large tech companies have been encouraging every organization that will listen to them to do for decades now

[throwaway3306a]

I have trouble accepting that. Even if they had to unplug the computer from the network and disable SSO and antivirus in safe mode, it's possible to get the computer operational. Even if they had to reinstall the OS and the critical software from scratch. There are solutions, the question is - did they even try? If not, why? And is CrowdStrike really to blame if they didn't? I just don't think so.

[advael]

Who in the org do you expect to have that competency, and do you think hospitals aren't keeping crucial things like credentials or software that gates access to things in the cloud when literally everyone in the world is encouraged to at every turn?

The culture of organizational IT is broken because a lot of powerful companies found it profitable to break it and leave something inadequate in its place

[WWLink]

I agree with this sentiment. If you ask me, the entity that comes out looking the worst from this Crowdstrike debacle are the companies that bought their service. Crowdstrike made a poorly designed and maintained product. I heard multiple people on reddit say it's the best of that type of product, but what the hell? Why does it need kernel-level control?

Why did we get here? If you're installing kernel-level software you might as well run a kiosk that only runs presigned code and runs off a read-only system image. And a lot of the machines in question DO APPEAR to be kiosk settings (like hospital data entry terminals).

It's easy to sit back and armchair, I'm sure there will be many cybersecurity experts who would figuratively jump at my throat for suggesting that trusting a vendor to run a rootkit on your computers is a bit incompetent. LOL. :D

[kmeisthax]

Everyone installing Crowdstrike seems like they want to build locked-down kiosks but haven't heard of Windows Embedded yet. Or at least I'm assuming there's an Embedded configuration that lets you do AMFI[0]-tier code signing enforcement.

[0] AppleMobileFileIntegrity, the daemon and kext on iOS that enforces very strict code signing.

[advael]

At this point I just assume any "cybersecurity expert" that defends Microsoft's nonsense is a cop

[throwaway3306a]

I expect the local admins to be able to install a fresh OS not connected to the enterprise network. And I expect them to have physical copies of stuff like disk encryption keys, also backups of OS installations and images, and all critical software. If they don't have that or can't use it during an outage, the problem is incompetent IT management that has no business running a hospital, not CrowdStrike. Something else would take them out sooner or later.

Again, we had all of this for a forest logging operation - is it too much to expect at a hospital?

[advael]

I agree with you, and kind of even agree that crowdstrike may not directly be at fault. But my point is that this competency is bled out of hospitals by external forces, primarily two: distant administration from companies that buy and manage multiple hospitals, often applying the same "efficiency" mindset that stripmines other industries in the name of profit, and the cloudtech sector, that is Google, Amazon, and Microsoft in particular, are very aggressive about selling their services along with demands that everything be given to their platforms, which often involves purging technicians who want on-site redundancy. This makes the systems more brittle, but also often causes people with the competency you're advocating to be fired

[lambdaone]

Absolutely. The risk being managed is the risk to the CEO/CTO's jobs, not the risk to life.

[toast0]

Hospital IT sucks. Look at a news report about a ransomware or this and it can easily be a few weeks for them to get back in shape. This one is hopefully easier because reportedly CloudStrike can sometimes pull an update before it BSODs and most windows machines auto restart on BSOD, so just leaving things unattended may be enough.

Restore from backup or reimaging fresh often means you need a working backup or image server, which at a lot of these places is also a Windows server and is likely also running the same endpoint protection, and is likely also boot looping.

Restore from zero isn't something any IT wants to do, and many of them aren't prepared to do it either.

Like it or not, hospital care revolves around the electronic medical records systems, and while Kaiser Southern California in the 90s was using amber screens and some sort of mainframe, afaik, almost everyone is on EPIC now, which is a windows application with all the baggage that contains. Even before EPIC took over Kaiser, they were running terminal emulators on Windows.

IMHO, it would be better for them to put together a ground up desktop distribution with exactly what they need, but that has user training costs and development costs.