[throwawaaarrgh]

I've worked on medical devices. As a techie, what surprised me more is not the lack of trials for new devices. It's that for software (and other kinds of components), it's often acceptable to have black box testing to certify a component as functional.

This means basically saying "I pressed a button and the thing did X". That's the test. No need to understand how that works, or provide any more technical documentation or specs, or record system state or variables, or do anything else. Press button, it does X. That's good enough to be certified.

The reason for this is simple enough: sometimes the thing you're using is proprietary and its manufacturer/vendor simply won't give you anything else to certify it. But this black box testing is even used for open source software and components. It's like a short cut you can use to cut through a lot of the testing you could have done. I imagine this will remain standard practice as the "AI" companies push their hallucinating dreck and unexplainable magic into the medical device space.

[neandrake]

I work with software medical devices. Another aspect here aside from 510k is that it is required, or effectively required, to comply with something like IEC-62304 (developing software as a medical device), and ISO-13485 (quality management systems for medical devices).

For the scenario you describe the piece that’s missing is risk analysis, a requirement. In preparation to release to market they must evaluate the probability and severity of the button not doing X or doing X incorrectly, and develop mitigations if the risk is unacceptable. What you ask - documentation and specs - exist at some level, but the manufacturer has to define what level is necessary for them. I could see an argument against the manufacturer deciding this for themselves, though it’s likely impractical to do so.

For software medical devices that have hundreds of transitive dependencies it’s not feasible to go at the level you’re describing. Some management of dependencies is necessary but treating as a black box - with quality/test management and risk analysis of the black box - is what the current system defines as a reasonable trade-off. Again I could see arguments for changing this, though for many manufacturers the EU has instituted stricter regulatory in the past ~5 years which has been a bit painful but overall probably a good thing.

Today one of the aspects of medical device development which is under tighter scrutiny is cybersecurity. It’s pretty painful right now. Previously there was not much related to cybersecurity required - obviously not ideal - but the pendulum has swung to the other end of the spectrum making it a significant burden. We’ll see, most of it is adopting new processes which is always painful and slows down progress at first. After the initial hump it should be eased into, and ultimately better for patient care and medical institutions in the long run.

[iancmceachern]

Agree with you here.

Wanted to echo " It's (cybersecurity) pretty painful right now".

The FDA just implemented new requirements. Basically they require penetration testing on all new medical devices. The issue is they don't have the expertise in house to know the technical details, and they haven't defined the tests, etc. Additionally they're isn't yet an ecosystem of partners and service providers yet to provide and compete in providing those penetrstion testing services.

Pragmatically what it means for folks trying to get a device cleared at the current moment:

You need to send your device to the one and only penetration testing house that does this for the FDA now and let them try to physically hack into your medical device. You have to make it impossible and evident if someone tampers with it in any way. This is in addition to all the software security stuff we need.

Imagine of you were making computer monitors. One day you are suddenly required to make it so a technical expert cannot open the monitor up using specialized tools.

[seehafer]

From a submission standpoint, as I write this, FDA seemingly cares more about cybersecurity than your medical device actually demonstrating safety and efficacy within intended use. The time that review teams have to review any given device has stayed the same, but fear-driven, heavy-handed cybersecurity regulations (which must be followed) have been added to the mix.

[NegativeK]

Respectfully, it sounds like the FDA is trying to implement requirements for manufacturers to do what they should've been doing all along. The shitty flipside to my point is that market forces pushed manufacturers to cut costs and externalize the infosec risk onto the patients. The secure products aren't interesting to medical healthcare providers.

I'm, admittedly, a bit salty because I recently looked at a healthcare device that I was prescribed and found evidence that my data is likely being trivially exposed by anyone who wants to look. I can't verify this because it's very likely illegal, and I don't feel comfortable reporting it to the device vendor for fear of being accused of hacking. If there's a way to report it to the FDA, I'd be thrilled -- but I don't know what that looks like.

[ivoflipse]

Did they provide you with Instructions for Use as a lay person? This might contain the legalese on what is or isn't permissible.

The company is required to have a complaint handling process, such that you making them aware of these vulnerabilities would mean they have to at least handle the feedback.

Maybe your findings can be rephrased in a way that don't require you to show how vulnerable their servers are, but that you suspect it's unsafe.

[iancmceachern]

True. The issue is they aren't specific about it. One penetration testing house may pass a device when another doesn't.

They haven't solved the issue. They've highlighted it and left it up to chaos to solve it.

[MichaelZuo]

I'm more surprised that medical devices were not previously required to be tamper evident.

[iancmceachern]

They are, and it's generally good practice. The difference is now they spend much more time and effort on the penetration testing.

An example. In the past it would be OK to use security bit screws for this. Yes you can buy the bits online, but it was at least one layer of perceived security.

This doesn't fly anymore.

The real challenge is they implemented these new requirements on devices that were already in the submission process. Also, these things aren't written down anywhere in standards etc. so you know them ahead of time when you design. You have to just wait until the penetration testing and find out.

Ultimately the new rules aren't the challenge, it's the fact that you don't get to know them when you start and finish the design, you find out later.

[MichaelZuo]

Isn't the intention that everyone submitting their devices for approval have done in-house penetration tests extensively? Or at least laid out specific claims as to what it can endure and what it cannot?

The third party test seems to be just the last verification stage to reassure the FDA the company is not making unsupportable claims.

[iancmceachern]

Not really because these things aren't defined.

There is no definition or standard to which you would do your in house tests to. It's not like other things where you design it to comply with iso whatever and then you test to that.

Here the standard so to speak is defined by the penetration test itself.

An example in safes. No safe is untraceable. Safes are spec'd by number of minutes to resist a tool attack. Then when a safe company goes to UL or whatever to certify the safe, UL technicians get the best commercially available tools and try there best to break into the safe and time themselves. If it takes them more than the spec, it passes.

Here there is no spec. There is no defined time. There is no standard. It's just up to what you can get the penetration test house to agree to write.

[jtwaleson]

It depends on the risk class according to IEC 62304. When you're developing Class C software you'll need acceptance criteria that goes into how you initialize variables etc.