[jmcgough]

I don't think you understand the scale of this problem. Computers were not up to print from. Our Epic cluster was down for placing and receiving orders. Our lab was down and unable to process bloodwork - should we bring out the mortar and pestle and start doing medicine the old fashioned way? Should we be charged with "criminal negligence" for not having a jar of leeches on hand for when all else fails?

[efitz]

I was advocating for a paper fall back. That means that WHILE the computers are running, you must create a paper record, eg “medication x administered at time y”, etc., hence the receipt printers, which are cheap and low-dependency.

The grandparent indicated that the problem was that when all tow computers went down, they couldn’t look up what had already been done for the patient. I suggested a simple solution for that - receipt printers.

After the computers fail you tape the receipt to the wall and fall pack to pen and paper until the computers come back up.

I completely understand the scale of the outage today. I am saying that it was a stupid decision and possibly criminally negligent to make a life critical process dependent on the availability of a distributed IT application not specifically designed for life critical availability. I strongly stand by that POV.

[krisoft]

> I suggested a simple solution for that - receipt printers.

Just so I understand what you are saying you are proposing that we drown our hospital rooms in paper receipt constantly. In the off chance the computers go down very rarely?

Do you see any possible drawbacks with your proposed solution?

> possibly criminally negligent to make a life critical process dependent on the availability of a distributed IT application

What process is not “life critical” in a hospital? Do you suggest that we don’t use IT at all?

[op00to]

Modern medicine requires computers. You literally cannot provide medical care in a critical care setting with the sophistication and speed required for modern critical care without electronic medical records. Fall back to paper? Ok, but you fall back to 1960s medicine, too.

[Wowfunhappy]

We need computers. But, how about we fall back to an air-gapped computer with no internet connection and a battery backup?

Why does everything need the internet?

[vkou]

> Why does everything need the internet?

Why would you ever need to move a patient from one hospital room containing one set of airgapped computers into another, containing another set of airgapped computers?

Why would you ever need to get information about a patient (a chart, a prescription, a scan, a bill, an X-Ray) to a person who is not physically present in the same room (or in the same building) as the patient?

[Dylan16807]

You wouldn't airgap individual rooms.

And sending data out can be done quite securely. Then replies could be highly sanitized or kept on specific machines outside the air gap.

[vkou]

You also need to receive similar data from outside the hospital.

And now you've added an army of people running around moving USB sticks, or worse, printouts and feeding them into other computers.

It's madness, and nobody wants to do it.

[galangalalgol]

This approach is also what popped in my head. I've seen people use white boards for this already so it must be ok from a hipaa standpoint.

[mandevil]

A hospital my wife worked at over a decade ago didn't use EMR's, it was all on paper. Each patient had a binder. Per stay. And for many of them it rolled into another binder. (This was neuro-ICU so generally lengthy patient stays with lots of activity, but not super-unusual or Dr House stuff, every major city in America will have 2-3 different hospitals with that level of care.)

But they switched over to EMR because the advantages of Pyxis[1] in getting the right medications to the right patients at the right time- and documenting all of that- are so large that for patient safety reasons alone it wins out over paper. You can fall back to paper, it's just a giant pain in the ass to do it, and then you have to do the data entry to get it all back into EMR's. Like my wife, who was working last night when everyone else in her department got Crowdstrike'd, she created a document to track what she did so it could be transferred into EMR's once everything comes back up. And the document was over 70 pages long! Just for one employee for one shift.

1: Workflow: Doctor writes prescription in EMR. Pharmacist reviews charts in EMR, approves prescription. Nurse comes to Pyxis cabinet and scans patient barcode. Correct drawer opens in cabinet so the proper medication- and only the proper medication- is immediately available to nurse (technicians restock cabinet when necessary). Nurse takes medication to patient's room, scans patient barcode and medication barcode, administers drug. This system has dramatically lowered the rates of wrong-drug administration, because the computers are watching over things and catch humans getting confused on whether this medication is supposed to go to room 12 or room 21 in hour 11 of their shift. It is a great thing that has made hospitals safer. But it requires a huge amount of computers and networks to support.

[WWLink]

> Pyxis cabinet

Why would a Pyxis cabinet run Windows? I realize Windows isn't even necessarily at fault here, but why on earth would such a device run Windows? Is the 90s form of mass incompetence in the industry still a thing where lots of stuff is written for Windows for no reason?

[mandevil]

I don't know what Pyxis runs on, my wife is the pharmacist and she doesn't recognize UI package differences with the same practiced eye that I do. And she didn't mention problems with the Pyxis. Just problems with some of their servers and lots of end user machines. So I don't know that they do.

[dghlsakjg]

You only need one link in the chain of doctor -> pharmacist -> pixys -> nurse to be reliant on Windows for this to fail.

[jmcgough]

This would be a disaster from a HIPAA perspective, and an unimaginable amount of paperwork.

[ikiris]

For relying on windows to run this kind of stuff and not doing any kind of staged rollout but just blindly applying untested kernel driver 3rd party patching fleet wide? yeah honestly. We had safer rollouts for cat videos than y'all seem to have for life critical systems. Maybe some criminal liability would make y'all care about reliability a bit more.

[dboreham]

Staged rollout in the traditional sense wouldn't have helped here because the skanky kernel driver worked under all test conditions. It just didn't work when ot got fed bad data. This could have been mitigated by staging the data propagation, or by fully testing the driver with bad data (unlikely to ever have been done by any commercial organization). Perhaps some static analysis tool could have found the potential to crash (or the isomorphic "safe language" that doesn't yet exist for NT kernel drivers).

[ikiris]

If you don't see that the thing that happened today that blew up the world was the rollout, I don't know what to tell you.

[nytesky]

A QR code can store 3 KB of data. Every patient has a small QR Sticker printer on their bed. Whenever EPIC updates, print a new small QR sticker. Patient being moved tear of sticker and stick to their wrist tag.

This much of patients state will be carried on their wrist. Maybe for complex cases you need two stickers. Have to be judicious in encoding data, maybe just last 48 hours.

Handheld qr readers, off line that read and display QR data strings.