[phren0logy]

I'm an MD, and I dabble in programming and AI/DL. While I don't really care much about Watson in particular, the pervasiveness of the "enter healthcare = death" equation is worrisome to me. And it should worry you, too.

There's a lot of potential for good applications of this type of tech in healthcare, but the barriers to actually getting it off the ground strangle it in the crib.

[criley2]

I work for a medical software company which evaluated Watson for use in our diagnostic product.

It should be seen as a Good Thing that medical is a Serious Field where Non-Serious Ideas get run through the ringer and fail.

And that's what happened with Watson.

The word from our engineers was that Watson was designed to sell commercials, not be a useful tool in a problem space like differential diagnosis or drug reactions.

I think this is good. Financial has the same reputation. You can't just use agile to bust out a minimum viable medical or financial product. You will be chewed up and spit out. And rightfully so. Because when it comes to our health, or our money and our investments, we don't have tolerance for failure.

OK, my social network double posted my communication, annoying but it's just a communication. However "my financial tool just double posted a transaction", or "my prescriptions tool just doubled the dose of a medicine" -- these are not OK states at all.

I wish Watson lived up to the hype, but sadly, our experience was that it does not. So it failing out of Healthcare isn't a surprise to us, nor is it a bad thing.

[megaman22]

> The word from our engineers was that Watson was designed to sell commercials, not be a useful tool in a problem space

This is the number one thing to remember about Watson. I've worked with teams within IBM that spent years and thousands upon thousands of man-hours trying to get it to be useful in a fairly well-defined, constrained use-case. Eventually they bailed and went to what was essentially a simple decision tree. It hits it out of the park on the sexy buzz-words, but I'm ready to curse the whole field for giving customers wildly inflated expectations.

[Spooky23]

I worked on a project that did a little Watson work as well. Same result as you described.

The shame is decision tree based expert systems can be pretty powerful, but the tools these guys were using were not that great.

[davak]

I am a nerdy physician desperate to work with somebody on decision tree based expert systems. Are there a bunch of existing projects that I’m just missing?

[gtmb]

Well, I just started an open source project to gather the data for such systems. It is starting as a database/graph of symptoms and conditions. But the vision is bigger than that.

https://github.com/gafmgafm/med

There are similar products but they are not open source, such as: http://apimedic.com/ and http://www.diseasesdatabase.com/

[mindcrime]

I've been doing a lot of work lately with automated Abductive Inference, and medical diagnosis has been cited as one of the key uses of same. Perhaps there could be some opportunity to collaborate (everything I do is open source as well, btw). If you guys would like to talk, feel free to hit me up at [e-mail]

[roymurdock]

Theranos is a counterexample. Probably would have been outed as fraudulent if they tried to enter a more safety-critical space (ventilators, pumps, injectors, etc.), but still a bad sign for medical diagnostic industry that it got so much funding and so far along, inking huge deals before collapsing. Perhaps some good will come if it makes others in the space more cautious in the short run

Next industry to keep an eye on (many are already) is automotive. There are a lot of vendors trying to put a lot of software/connectivity into the next generation of cars. Lots of regulation but I will opt for a dumb car over new smart stuff any day.

Same with a lot of different traditionally embedded systems that are now being connected to the internet

[nimish]

No, Theranos started to fail as soon as anyone with real medical expertise without a stake in the company started to probe it.

Theranos was extremely successful at fleecing VCs and embarrassing political board members.

It's a good thing that it got so far, because it proves just how bad VCs are at evaluating successful companies and how much of the funding game is who you know.

[abakker]

Arguably, the big deals and optimism indicate that there is a lot of pent up demand for successful things, and people are willing to take some risk to get solutions. It’s not all doom and gloom, solutions just have to work.

[sk5t]

Conversely, I read Theranos's progress as evidence of cynical bets that the government/military could likely be directed into pumping billions of USD into the company regardless of its merit.

[abakker]

You could certainly draw that conclusion from its original board. I did as well. Still, I think the point stands that people want healthcare to work, and will pay for it. It just so happens that it takes a lot of effort to improve the system since what we have is pretty good. (Not to say it doesn’t have problems)

[phren0logy]

Oh, I agree if it doesn't work it should be tossed aside. My concern is that given the potential usefulness demonstrated in so many other domains, I suspect that many of the tech failures in healthcare are due to the fact that it's too expensive to take the time to get it right.

[mlthoughts2018]

I think this might also be supported by looking at some of the better funded / successfully exited health tech startups.

I’m thinking of things like Flatiron Health, BenevolentAI, Quartet and Spreemo, and surely many more.

At heart these companies seem to shy away from being “really about” medicine, and try to be a more watered down data company, looking at health records, social media, and other data to provide recommendation, service matching, comorbidity analysis, hospitalist tools, cost tools, assistance program allocation tools.

I’ve been very interested in ML applications in health, but a lot of the businesses that seem capable of getting funding seem like they are super light / elementary on the statistical modeling side, and the value add is just a claim to modernize crappy claims data or unify a bunch of previously disparate health data sources.

Maybe more advanced use of modeling will come later, though I am skeptical just given the way that once big enterprise customers dig hooks into essentially consulting services for health records, they won’t let go.

Another thing that creeps me out is when you see e.g. an ex Palantir board member joining health record data companies. It’s not hard to understand why a big insurance company, or at worst even government agencies, want decision tools on top of huge stores of health record data. Do we really benefit from some super new / unproven startup building up that data set & tooling?

[sergers]

I had posted this previously in a different thread, but it's all smoke and mirrors.

I work in the diagnostic imaging space, we are working on integrating AI for assisted diagnosis or screening. It's a several-year road map for some base function.

We had a top sales executive leave for IBM Watson Healthcare division about a year ago, was very proud to be part of what Watson was going to bring to healthcare

He's been working back at my company, couldn't stand selling lies (and wasn't making much commission with canceled contracts).

From a named company more impressed with what Google deepmind has done with NHS

[yosyp]

Tangentially related: how does one get their foot in the door at a medical software company? Is there a big market for such professionals?

[pram]

Learn MUMPS and move to Wisconsin.

[walshemj]

Is MUMPS still a thing :-) maybe I ought to dust of my RT-11 / FORTRAN skills off

[prepend]

The biggest EMR company was Epic and uses Mumps [0]. However, they are losing share to Center and others [1], so this formula will only work for a little while longer.

[0] https://news.ycombinator.com/item?id=13860937

[1] https://ehrintelligence.com/news/cerner-epic-mckesson-among-...

[sergers]

Epic still dominates the EHR space.

They have deployments slotted for 3 years.

If you have ever worked with epic, they offer "managed" solutions, which you either do the epic designed way, or don't go epic... They have enough market share and business to dictate how things will work in a hospital env

[dbs]

You get at the door of companies in the healthcare space that need to get up to their game and tell them what you can do for them. That's what I did.

[tjr225]

I was recruited right out of college to work at Cerner. Its probably still fairly trivial to get a job there, but you have to live in Kansas City.

[bitwize]

I think this has more to do with the fact that Watson sucks than anything else, although it may be a harbinger for other efforts to sprinkle magic machine-learning pixie dust on complex problems in the hope for a solution.

[phren0logy]

In this case, you are probably right. But it seems to happen over and over again suggesting that the barriers to innovation are higher than they should be.

[serioussecurity]

Yeah, to be clear, Watson failing is good news. This is marketing people losing their minds because they didn't understand the products they were selling, and hospitals failing to do due diligence.

I've talked to literally dozens of people / companies / academics running at problems in healthcare using deep learning and large data sets, and the recurrent theme is cowboys who pretty consistently fail to follow basic practices of experimental rigor. ML is frankly an area where more rigorous and clear regulation is desperately needed, because the potential to cut corners / cheat is so easy and the challenges of vetting these systems are specialized enough that most organizations won't successfully be able to catch fairly basic experimental mistakes.

[ISL]

Many of those barriers have historical reasons for existing -- which do you think could be lowered without risking patient health (or yielding sufficient benefit to be worth some added risk)?

[phren0logy]

Probably. The following applies to the US: It seems to me that some of the biggest barriers involve getting access to enough data to make a good training set. Some of this is related to HIPAA and related privacy laws, which are in place for a good reason but are still a major barrier. The other big factor is the fragmentation of the data across different, non-interoperable, non-standard formats from different vendors who have intentionally made interoperability difficult. That part bothers me much more. Many patients have data spread across a dozen or more paper charts, lab systems, EMRs, pharmacy databases, etc. As with most data science tasks, the "data munging" is the hardest part.

I care about this mostly from the perspective of treating patients. Beyond training sets for AI, I need those records for the same reason: to make appropriate and informed treatment decisions.

From the outside looking in, it's the kind of situation that begs companies to over-promise on results to get enough money to even give it a shot.

[aaavl2821]

Unfortunately, from my experience the biggest barriers to sharing data are very deeply embedded incentives.

I was at a talk recently by an executive in charge of managing data for a massive health system. Think tens of millions of patients. He said they dont share data because if people learn their real quality metrics it would give payers, the public, anyone with an interest that isnt aligned w the hospital system ammunition against them. Have heard this sentiment echoed by many. Open data is their enemy because then patients could choose based on outcomes, not on market power or how nice the lobbies are

The EMRs designed a closed garden into their systems DNA. If data is shared freely then it's easy to switch systems, and they will have to compete based on cost and quality and couldn't charge hundreds of millions for installations and maintenance contracts

Call me cynical but this is all based on conversations with people well placed in the industry

[xoail]

I am working on a medical dictation app and the lack of any interoperability between 1000s of EHRs is mind boggling to me. There is absolutely no standards and like you mentioned its completely deliberate. Working through these platforms over the years has made me less and less optimistic about having a better health care system in the future.

[sergers]

Well FHIR is making its way, that looks positive, but no mainstream EHR has adopted it really.

May I ask what your target users are? I sell alot of nuance PS360, and fluency

[ISL]

As a patient, I'd love to be able to archive/hold my own comprehensive copy of all of my health records.

[jessriedel]

Airplane security has a historical reason for existing too, but that doesn't mean the benefits justified the costs, either now or in the past.

[Thriptic]

The two massive barriers to entry are regulatory uncertainty and sales difficulties

Regulations: Up until recently, the regulatory landscape for medical software was extremely confusing. People were sitting on the sidelines waiting for FDA to figure out how to deal with machine learning, agile development, app stores, blah blah. Startups that wanted to jump in were staring at yearly 100K+ regulatory consultant bills. FDA is now actively de-regulating the bulk of medical software, and will create a regulatory kit which should better explain the regulations in an actionable way (instead of "create a design history file" without much explanation of wtf that means in practice). This should ease pains on that front

Sales: This is where the big pain is. Getting paid for shit in healthcare is just too damn hard unless your product is on the hospital operations side of things. Getting an FDA approval (assuming it's required) DOES NOT mean one of the thousands of insurers will decide to reimburse you for your product; many require multiple clinical trials for reimbursement purposes; and many will decide to stop covering your product for stupid reasons (ie some editorial in the "No One Reads This Journal of Nonsense" said it was unclear the product was effective). Good luck getting patients to pay for anything, as they expect all products to be covered by insurance. Good luck convincing doctors they should use your product, as they are already overwhelmed with data entry / shitty UIs and don't want to perform any additional clicks or workflow changes.

[criley2]

The FDA is not the primary source of regulation in medical software, it's CMS. And while CMS regulations do not apply theoretically to private insurance and private practice, Medicare and Medicaid represent such a large segment of the healthcare economy that private insurers tend to use similar expectations as CMS regs.

Also, software which cannot meet CMS reg and cannot be used for Medicare/Medicaid patients is a non-starter for the majority of the healthcare industry.

The idea that our medical software space would be mostly deregulated is a rather horrifying thing in and of itself. I don't want my father being cared for using beta, buggy, software. I have high expectations for quality and risk management from healthcare software that I do not have for some agile beta video game or social network.

I think your "bulk deregulation" comment is rather wrong. If you are coding in this space and are not INTIMATELY familiar with the concept of PHI and HIPAA, then you will be writing software which violates core regulations in this space.

Your concerns on the sales side also do not mention "RISK" even once. You don't speak even a little bit of our healthcare space. Those doctors want risk management. What happens if/when the software screws up? Whose fault is it that the software is literally response for killing patients? You also don't mention what doctors care about: does it meet CMS regs? Does it meet state regs? Will I be able to submit my obscure XYZ state form? Will I be able to submit to BCBS, Aetna, Medicare? Will they accept? Etc etc. You will be writing custom software for customers all over, because the regulations that states and even cities like NYC might impose could be imposed on a single customer of yours only.

I think people who want to disrupt healthcare forget that healthcare IT is literally life or death. Your bug isn't just an annoyance, it could be the thing that ends someones life. Regulations aren't just a barrier to agile development, they're also a life saving tool to ensure that risk management and data privacy are established in all products that get used on patients in our country.

[Thriptic]

> The FDA is not the primary source of regulation in medical software, it's CMS

It's totally dependent on what you are doing. Sure, HIPAA / PHI concerns are the primary issues for people who generally make software for the medical space, but by "medical software" I mean software that FDA might classify as "software as a medical device" AKA FDA regulated software. If your software falls under this designation, as most diagnostic medical AI systems like Watson would, then your primary regulatory concern is the FDA and you are held to much more stringent software development requirements than someone that has to deal with some PHI concerns.

In this context, my bulk de-regulation comment is 100% correct, as the FDA has basically dictated almost everything in the space to be unregulated (all back office products, general wellness products, enforcement discretion products, MDDS, even CDS which almost certainly should be regulated). They have literally gutted Class 1. You basically have to be writing something that is going to provide a diagnosis or treatment plan, control an existing medical device remotely, or mimic an existing regulated medical device in functionality in order to be regulated now. Reasonable people weren't scared to enter the space because they might have to deal with PHI for HIPAA concerns, as it is well known how to do that at this stage; they were scared to enter because there was (and continues to be) some uncertainty as to how novel product classes will be regulated by FDA.

> Your concerns on the sales side also do not mention "RISK" even once. You don't speak even a little bit of our healthcare space. Those doctors want risk management. What happens if/when the software screws up? Whose fault is it that the software is literally response for killing patients?

All FDA regulated software is mandated to have robust risk management planning prior to approval (ie ISO 14971 compliance). Other than that, the vast majority of products that a patient will be interfacing with can't do them real harm if they are buggy. That is precisely why the FDA has chosen not to regulate them.

> You also don't mention what doctors care about: does it meet CMS regs? Does it meet state regs? Will I be able to submit my obscure XYZ state form? Will I be able to submit to BCBS, Aetna, Medicare? Will they accept? Etc etc. You will be writing custom software for customers all over, because the regulations that states and even cities like NYC might impose could be imposed on a single customer of yours only.

I don't mention doctor wants because doctors aren't the buyers most of the time. Payers are the buyers, hospitals are the buyers, or patients are the buyers. How many pieces of software that you know of are sold directly to the doctor?

[sergers]

My company software is considered a Class 2 medical device.

We have to go through so many crippling hurdles with FDA it's almost not worth it.

You are spot on.

To add though, In atleast the diagnostic imaging space, radiologist reading groups/practices are selecting their own systems more and more nowadays.

Individual doctors? Likely not.

[616c]

So why do you need FDA regulation in your situation? I know people handling security regulation in the defense industry and I'm curious how you know the FDA has to approve you as opposes to CMS and what it entails.

[sergers]

For diagnostic imaging (x-rays, CTs etc) which we make software for, alot of equipment is FDA regulated.

We chose to go for class 2 designation in FDA terms as it shows you comply with uptmost regulations, which is a selling point. Some hospitals depending on state gets rebates for meeting compliance.

It depends on what space you are going into. If it touches a patient, more than likely FDA regulated in the USA.

Other countries have similar orgs ( Canada has... Health Canada).

We also work with DoD/VA... Don't get me started on compliance with them.(the security stuff is a plus though, while initially challenging to meet, it has us pushing the code into commercial space products, everyone benefits from the extra security hardening. )

[Thriptic]

What it entails depends on the product. FDA has 2 regulatory classes which require a premarket notification, class 2 and class 3. Class 2 submissions generally require you to comply with a set of software development standards governing testing and validation, requirements generation at multiple levels, interface design, risk management, traceability, design, and actual development. You are required to handle customer complaints, safety problems, and maintenance in specific ways. You are required to produce customer facing documentation in specific ways, have installation plans, etc. Finally, you need to register yearly with the FDA and have all sorts of training and infrastructure planning. Proof of all of this is to be constantly generated (think every test report) and it all gets put into a massive file for FDA review. There are also some writeups you need to do explaining why you are class 2 as opposed to class 3. This generally involves finding a class 2 piece of software which resembles yours and saying your software is basically doing the same thing. This is typically referred to as a 510(k) submission.

There is also a pathway you can apply for if there is no existing device with a class 2 designation which resembles yours, but you believe your device is still fairly low risk. That is called a de Novo submission.

A class 3 submission generally involves even stricter requirements for development plus a clinical trial to prove efficacy and safety. The most common one if these is called a PMA which stands for premarket approval.

Also, if you are FDA regulated you still have to comply with CMS regs. It is an extra layer of regulatory scrutiny.

[phren0logy]

Glad to hear things are improving on the regulations front. Sales is going to be hard for a while, but I think that there are some markets where this is less of an issue, as it's more cash-based anyway. It makes me sad, though, that a lot of the population-based preventative stuff for which AI would be a great fit is unlikely to be a financially appealing investment.

[killjoywashere]

Sales: skip it. Cardinal, Leica, Stryker, and others already have sales workforces. Cut your deal with one of them instead.