[11thEarlOfMar]

/rant/

I can't tell you how many times we've chased down field problems that ultimately were the result of antivirus scans. It's been so bad, that one of the first questions we now ask when we get a tool-down report is "is there antivirus running and what is the configuration?"

Bringing Windows into the architecture of any type of capital equipment control system is a bane. A scourge. I mean to say, it really is a misappropriation of software. Imagine, "Yeah, Frank only knows VB, so that's what we used for the aircraft's cockpit GUI."

/xrant/

[raverbashing]

This

This machines costs hundreds of thousands of dollars.

There should be no excuse for using Windows. None.

I would not be surprised if the "antivirus" thing was some PHB requirement

[Amezarak]

Is there a reason to believe that choosing Windows was a bad decision?

The bad decision was installing antivirus software. Otherwise, most any modern OS would be fine. This machine probably shouldn't be connected to a network (if it was), the USB ports should be disabled; data can come off on burned CDs, autorun should be disabled, etc. That's how you deal with IA concerns on a standalone mission-critical system, not by installing antivirus.

[ChuckMcM]

I think you have it backwards (no disrespect intended). When you evaluate the choice of Windows you have to acknowledge that it brings with it the vulnerability of viruses and so the necessity of anti-virus software. Either you own that decision, and as part of your support your tool provides the necessary antivirus and you also insure through testing configuration management that its configured appropriately, or you choose a different option up to an including writing your own system to manage the "time critical bits".

Having their software run in a Windows ecosystem that they do not have strict configuration management control over was a bad decision and on the basis of this failure report. That it did not result in patient injury or death was fortunate but is certainly not guaranteed.

[Amezarak]

> When you evaluate the choice of Windows you have to acknowledge that it brings with it the vulnerability of viruses and so the necessity of anti-virus software.

No matter what OS you choose it is vulnerable to viruses. You and I will agree that the odds are your Windows system is much more at risk by at least an order of magnitude. But the IA people who demanded that this system run antivirus are just as likely to demand that Linux run antivirus, simply because the vulnerability theoretically exists and making that demand fulfills their CYA requirements. I've worked on standalone Linux systems that IA demanded have antivirus.

> Either you own that decision, and as part of your support your tool provides the necessary antivirus and you also insure through testing configuration management that its configured appropriately, or you choose a different option up to an including writing your own system to manage the "time critical bits".

According to the article, the software runs on the user's hardware. While they certainly could have made a decision to provide their own controlled hardware, it's entirely possible that hospital was not open to that option for cost reasons, for IT management reasons, whatever.

[ChuckMcM]

   > No matter what OS you choose it is vulnerable to
   > viruses.

It is a bit more nuanced than that. While it is true that absolute security is generally deemed impossible, if you use a widely deployed operating system in your device there are both a number of actors trying to compromise it for different reasons, and a number of examples that can be acquired for testing different exploits.

By writing just enough "OS" to achieve your goals in an embedded system, and then designing a clean access API through which you cannot affect the underlying code (no "here download this new firmware" call) you can avoid that particular threat vector.

[DyslexicAtheist]

> You and I will agree that the odds are your Windows system is much more at risk by at least an order of magnitude

it is a common misconception that Windows is still worse a platform than Linux when it comes to security. Not trolling ... I'm using Linux since '96 and built my life and career on it. Opinion of some people in infosec circles (@thegrugq @csoghoian ...) is that Windows no longer lags behind:

https://grugq.github.io/presentations/COMSEC%20beyond%20encr...

[dantiberian]

There's two separate things that are conflated here:

* Is Windows security equal to or greater than Linux and OS X (Probably yes)

* Are the overwhelming majority of viruses written to target Windows systems (yes)

I really enjoyed this article on security economics which goes into this in more depth: http://tidbits.com/article/15939

[wyager]

>No matter what OS you choose it is vulnerable to viruses

This may be theoretically true, but it is not practically the case. There is a reason Linux and OS X users almost never use antivirus software.

Using Windows on a medical device is inexcusable. It's a heart monitor, not a game system.

[clay_to_n]

Many big-name vitals monitors seem to run Windows under the hood. They have a whole PC in there, connected to their hardware sensors. If you pay extra for network connectivity or a similar premium feature, I suspect it's the same software with some flag turned on.

[Coincoin]

Medical devices are used by humans. Humans, for some reason prefer and know Windows better. Why shouldn't the devices use Windows?

Today, doctors from a hospital the other side of the country can diagnose your cancer realtime while you are still in the MRI machine, and they do it with Windows, because that's what humans know and use.

[nickpsecurity]

"No matter what OS you choose it is vulnerable to viruses. You and I will agree that the odds are your Windows system is much more at risk by at least an order of magnitude. But the IA people who demanded that this system run antivirus are just as likely to demand that Linux run antivirus"

That's not true for what those systems should be really running which is separation kernel platforms. These isolated tasks in partitions using high-assurance kernels designed to not fail in every way you can think of and with almost no code to hit. The apps even donate their own resources for kernel calls. Interface is in untrusted VM that sends checked commands to real software running on isolated partition optionally on Ada or Java runtime for memory-safety. Anti-virus is not available and not necessary given the untrusted part is strongly contained & the trusted part is memory safe.

http://www.ghs.com/products/safety_critical/integrity-do-178...

https://os.inf.tu-dresden.de/papers_ps/nizza.pdf

Just a matter of using right tools for the job. Any medical device using Windows or any other complex OS isn't doing it right. Even OpenBSD would've been a better choice given it rarely is hacked, crashes, or needs updates. Antivirus software wouldn't run as it's not available for these. It would be a network appliance or something that didn't affect running system.

[Diederich]

Is the day to day impact of antivirus on Linux the same as on Windows?

[Cheapshot1]

even better analogy is. Is the day to day impact of your own customized OS the same as windows. If the medical equipment ran independently we wouldn't have anything really to worry about.

[raverbashing]

You're right. My bet is that they use windows because they want to save or get something from the machine (either to a USB drive or network)

And I would have followed the same steps you mentioned. The machine could work with Windows.

Windows CE might have been a better pick, so you can have things like RO filesystem, etc

"There is no reason to use Windows" is, as you mentioned, not a bad decision in itself, but since they do it the lazy way, it is awful. Not sure if there's an out-of-the-box way of firewalling everything in the Windows versions available at the time that machine was built

[x0x0]

I worked on similar software that ran on Windows, though the decision was made 18 years ago. It was driven by:

* GUI dev tools were an order of magnitude better than under linux

* nidaq (very common daq and dio cards) didn't have or had flaky linux drivers. usb may be a better choice today but was not then available. Plus it's convenient/less expensive to push functionality to the computer side and minimize embedded development.

* in the beginning (though we fixed this mistake), we shipped software, our machine, and a ni-daq card; users provided a computer. This was a nightmare. We ended up shipping whole computers custom built by Dell to our specs with ni-daq cards installed (and later glued in to prevent them slipping out during shipping) plus our custom system images. At the time, I don't think Dell would do this for linux; you could find other vendors that would, but they weren't easy to work with or much more expensive.

* Nobody is eager to switch OSes because every software update to the machine has to be FDA certified (ie kiss at least $100k goodbye), so the company is very risk averse. Beyond even the cost to rewrite what has evolved into a relatively complicated gui app.

* As mentioned elsewhere, once you leave windows you have to train people to use linux guis that historically had window managers / gui toolkits that didn't work like windows.

All of the above is complicated by the fact most of these medical device companies pay their devs shit and have trouble hiring good devs. (Engineering shortage, obviously!)

[tremon]

every software update to the machine has to be FDA certified

Interesting. Does that imply that every Windows update Microsoft releases has been FDA-certified already?

[icegreentea]

I don't know how it works out for stuff used during surgery, but for diagnostic devices (where if a device fails by becoming inoperable, the stakes are a lot lower), you can lump basically any software that you don't have total control and design history on (so this includes random proprietary software and most FOSS stuff) as "software of unknown providence" (SOUP).

Now based on the class of medical device you are developing, you can either get to use SOUP without justifying at all, or you have to explain/justify why you can use SOUP at all.

For example, even if your OS was Linux, because you don't have formal documentation on verifying that Linux infact does all the things its suppose to do (hell, does Linux even have a true spec?), you would then have to justify why you're using Linux. If a kernel update comes out, you then have to justify why you're using it.

What you WOULD have to do is verify/validate that your software that interacts with SOUP (so in this case running on an OS) still works correctly. You would not have to verify/validate the OS update per se.

[HeyLaughingBoy]

His characterization is not quite accurate. FDA does not certify it; rather, the company building the medical device certifies to the FDA that it has been properly validated for use and the FDA accepts their validation.

Trust in the company's ability to properly validate the device is tempered by regular FDA (and other regulatory bodies) audits of their development process and process data, test reports and a general desire to stay in business. FDA can and will take your product off the market if it appears to be unsafe and the company does not respond appropriately.

[enraged_camel]

A lot of those machines have Windows embedded in them, because a lot of doctors and nurses are really, really bad with technology and cannot use anything more complicated than Windows's familiar user interface. Training involves "OK, now use the mouse and double-click this icon on the desktop to start the program."

I wish I was kidding.

[afarrell]

> I wish I was kidding

I know that "tired person in a hurry" is a trope, but considering how masochistic the health professions are about sleep, it really applies here.

[stephengillie]

Windows is a familiar interface for computers. Is there a good reason for not using a familiar user interface?

[tremon]

Huh? Of course there is. "Familiar interface" isn't even a functional requirement, it is only of secondary importance for any system. Actual functionality is much more important.

As an aside, if "familiar interface" is your only requirement, I'd suggest to install a door handle or light switch.

[et2o]

Surprised I had to scroll this far to find someone blaming a physicians.

[jcrawfordor]

I think a lot of people are misreading this article. It appears that the device is standalone and runs some embedded OS, but as a feature logs data to the doctor's existing computer. So it's not so much that they built the tool on windows as it is that they built the tool on the operating system they expected doctor's to be using.

I still think this was a poor decision, but it's a different kind of poor decision

[chapium]

To be honest, there is an excuse, although not a very good one. The machine is costly because it is in high demand. R&D and pushing this through regulatory control are costly. This company has great incentives to cut costs to remain competitive.

[heartbreak]

Perhaps the regulatory control should cost more.

[ArkyBeagle]

Windows ( at least 7 & XP) can be stripped down to something approximating a properly running system. This isn't easy.

Having an anything-critical Windows machine on the open Internet might not be such a wonderful idea.

[copperx]

I would expect these kinds of systems to be running a soft realtime OS. Or at the very least a run of the mill OS with no extraneous software running in the background.

[Dwolb]

This. How are these devices not running on some sort of hardened OS seen in airplanes and automotive? Medical applications are mission critical (or some variant) and should have same (or better!) certification procedures set up for correctness and security.

[aavotins]

Second this. It is terrifying to know that mission critical, medical grade software runs on a consumer operating system. Military/aerospace systems have numerous requirements and clearly defined practices and ways of developing these systems, often going through various layers of documentation and using specifically designed programming languages(like the Z programming language) to write specifications, which are then re-written into code, but it seems like medical industry has been neglected.

[ktRolster]

Then this story about viruses at a nuclear power plant won't make you feel any better: http://www.reuters.com/article/us-nuclearpower-cyber-germany...

Some great quotes:

"Mikko Hypponen, chief research officer for Finland-based F-Secure, said that infections of critical infrastructure were surprisingly common"

"Hypponen said he had recently spoken to a European aircraft maker that said it cleans the cockpits of its planes every week of malware designed for Android phones. The malware spread to the planes only because factory employees were charging their phones with the USB port in the cockpit."

[zyxley]

> The malware spread to the planes only because factory employees were charging their phones with the USB port in the cockpit.

The moral of the story: If you include any kind of port in something, people WILL plug things into it sooner or later.

[ams6110]

Sounds suspect to me. Aircraft computers are not running Android.

[copperx]

I'm pretty sure the entertainment system runs on some version of Android.

[clay_to_n]

Is there Android malware that, when connected to a Windows PC, spreads Windows malware? Sounds reasonable in this situation.

[koluft]

They run USB, it doesn't matter what OS is on the other side.

[radicalbyte]

I've been working in Medical for a couple of years... it's because Medical is extremely fault tolerant. They put up with a lot of rubbish that wouldn't be accepted in other industries.

Aeronautical and Automotive are both engineering driven, Medical isn't, it's a big grey area.

[wyldfire]

You might be surprised to realize that many applications of medical devices are not used for affirmative life support and therefore should not be held to the same standards as aviation.

This application (cath lab activity logging) is not a life-support activity. Product failures of any kind (either due to design error or product defect) represent a diminished capability of diagnosis and treatment. This does not represent a risk of harm to a patient.

That said, some medical device manufacturers treat this aspect of design very seriously and go to great pains to use defeatured and heavily restricted OS and settings.

[beachstartup]

correlation doesn't imply causation.... except when someone selected windows, and then made 50 other moronic decisions also.

half the work done in this industry is just dealing with stupid decisions made by stupid people. i just accept this now.

[tluyben2]

But when Windows is chosen because it is easier to find coders who will 'remain within the budget' (cheapcheap) and cannot even make sure the virus scanner doesn't run during procedures or at all it goes a bit too far.

[Amezarak]

According to the article, it was the hospital IT that misconfigured the antivirus, not the application developers.

Is there some reason to believe this wouldn't occur on a Linux system? There are plenty of dumb IA requirements that antivirus be installed on Linux, too.

[zanny]

Hardware involved in heart surgery running a Linux system should have literally the bare minimum of software to run it. That means kernel, init, networks support (dhcpcd, systemd-networkd, if absolutely necessary samba, sshd, etc) and custom software running on top of that as nobody. Not some ~20GB of Microsoft crap driving the system to crash every couple days.

[Amezarak]

> Not some ~20GB of Microsoft crap driving the system to crash every couple days.

I agree the system should be running as few services and as little other third-party software as possible, but let's be fair. Since at least Windows 7 / 2008R2, particularly for an offline system, the OS is not going to crash unless there is a hardware problem. It's not clear the OS crashed even in the article - "the screen went black" (the application went black?) and they "had to reboot" doesn't give us enough information.

A modern Windows system, like a modern Linux or a modern FreeBSD, is stable and will stay up for as long as you need it to, unless as I said before, there is a hardware problem. (Or in the case of consumer Windows, you do an update.)

EDIT: According to the actual report, the OS was not rebooted, the application was. There was no Windows crash.

> On (b)(6) 2016, a customer reported to merge healthcare that, in the middle of a heart catheterization procedure, the hemo monitor pc lost communication with the hemo client and the hemo monitor went black. Information obtained from the customer indicated that there was a delay of about 5 minutes while the patient was sedated so that the application could be rebooted. It was found that anti-malware software was performing hourly scans. With merge hemo not presenting physiological data during treatment, there is a potential for a delay in care that results in harm to the patient. However, it was reported that the procedure was completed successfully once the application was rebooted.

[pdkl95]

"Cheaper" or "remain within the budget" doesn't excuse using inadequate parts that don't meet the design requirements.

Unfortunately, this total disregard for safety isn't just software anymore. When we stat skipping lessons that we've know for a looooooonng time (such as why a split bobbin is an important feature in a transformer[1]), we have evidence of a serious need for strongly enforced regulation.

[1] https://news.ycombinator.com/item?id=11474730

[tluyben2]

I know it doesn't, but unfortunately it is often how it works. More than I want to remember I have seen things like 'but lives depend on this!' or '100s of millions can get lost if this doesn't work!' and yet when the RFPs come back and something like InfoSys is chosen because big name and cheaper than experts in the field.

Edit;

> we have evidence of a serious need for strongly enforced regulation.

Better education? But I guess strongly enforced regulation will force companies to not go for the cheapest solutions they can get away with which in turn will require people with actual knowledge in the field which will require better education, somehow.

[pdkl95]

Better education is always a great idea. Unfortunately, regulation becomes a necessary fix for immediate problems.

Note that regulation is the nicer option; the other way to force people to get the necessary education is liability, which could get really ugly in the case of medical devices.

[tluyben2]

There have been 'predictions' in the past of software creators being made responsible for the software they write in a liability way. Which will get very messy indeed. And grind the software world to a halt. Regulation is the nicer option and definitely the more realistic option.

[makomk]

Sometimes it's all about tradeoffs. For example, a split bobbin is actually undesirable in most transformers because it reduces coupling between the windings, reducing the efficiency of the power supply. That's why if you look inside a better-quality power supply you'll often find that they have a split bobbin for the input common-mode choke and then a single bobbin with layered windings for the main transformer.

[dredmorbius]

Only half?

I mean this quite seriously: you're hugely underestimating. Possibly unaware.

[rsync]

"Bringing Windows into the architecture of any type of capital equipment control system is a bane. A scourge."

We need to go much further than this, since many people will "solve" this problem by using a different platform than Windows.

In reality, bringing networking of any type into capital equipment control systems or critical infrastructure is the bane ... the scourge.

Whatever convenience or perceived function that networking (including very local networking, like USB) is dramatically outweighed by the additional attack surface.

Go back to sneakernet and check your facebook at home, Mr. Nuclear Plant Worker.

[CaptSpify]

The issue with this is: Many of these systems need to send data to other system on the network. You'll have to send the data over some kind of PACS network, or you'll just have to use a USB.

I agree that's a "better" setup than networking, but I still don't think having staff plug random usb devices into your medical equipment is a great idea either.

[rwmj]

And if you want PCI-DSS [credit card handling] certification, then you'd better be running AV software, even when it's completely inappropriate.