[burnte]

Healthcare CIO here. This is true. Healthcare is still using paper fax. It has a 30 year old data interchange format that no one really supports because it's more profitable to lock in customers to your EMR. Healthcare is HORRIBLE about upgrading anything, at changing processes, and technological progress in general. Healthcare is VERY backwards from a tech standpoint.

Another problem is that EVERYTHING is custom, we use very, very few off the shelf solutions. Need an EMR? Let's build it in MUMPS, a 51 year old language that originated on the PDP7 and call it a state of the art system like Epic or GE Healthcare. Don't like the terminal interface? Let's slap a GUI on the front that still interacts via TTY on the back end. SQL? Nah. C, C++, or any more modern language with more robust features and way more programmers? Nope.

Now, there are some EMRs and other healthcare-centric apps that are better written, but they're also terrible. Healthcare is a relatively small market, you'll never sell a million units of your app, so you charge out the wazoo for it, get a few health systems on it, and allow they to go crazy with customization to help lock them in. And then you try to add on modern security features on to a system that's been growing for 50 years and it's a nightmare. It's INCREDIBLY common for nurses and doctors to need to have administrator access on their Windows desktops for various apps.

I was about to leave IT in general when a healthcare gig landed on me, and I'm glad it did. I find it very refreshing to be in an industry where it's so far behind that there are mountains of problems to tackle, even if half of them are so stupid it makes me want to cry.

[rubatuga]

People need to stop hating on fax. Hospitals still use fax because it is a much more punishable crime to tap phone lines which requires physical access, as opposed to a server that could be infected from a hacker halfway across the world.

[keithnz]

Fax is odd, it was a fantastic thing when it first came about, and it has some desirable properties.

- It's direct point to point communication (over a network)

- The transport network is dedicated and not open to anyone and covered by quite strong laws in many countries

- It's easy to see the history of communications

- It's easy to see if the other end successfully received something

- It's relatively standardized and ubiquitous ( in health )

Email would be the closest thing, but it doesn't have all the advantages, and the extra add ons that would make it better (like encryption, delivery receipt, digital signatures) are not standardized and/or ubiquitous ( and often hotly argued about )

So fax is the lowest common denominator, that, if it was proposed today, would not be accepted for many of its disadvantages, but it's now hard to find a way to replace it.

[gruez]

>- The transport network is dedicated and not open to anyone and covered by quite strong laws in many countries

is it? what if the hospital is using a VOIP solution?

[dredmorbius]

...when it first came about...

[analog31]

- It's easy to see if the other end successfully received something

I think this is a biggie. It means your workflow doesn't need to include going back later and checking to see if your document was received, and then trying to send it some different way. You don't have to guess which way the recipient is capable of receiving a message.

It's the original e-mail. ;-)

[XaspR8d]

Except seeing it was digitally received is often quite insufficient to seeing it was received by a human it was intended for. All too often in dealing with healthcare and gov't orgs our faxes get lost with no way of identifying where they went. Presumably it is a mismanaged shared fax inbox where individuals are not actually being alerted to their messages...

[keithnz]

somewhat ironically, the fax will be captured in digital form, some "middle" person will read it, then work out who it is for, then email it to those concerned / attach it to a patient record.

[DEADBEEFC0FFEE]

No quite, many fax system are just modems, image conversion to pdf, email. Plenty to go wrong. Paper faxes might be revived by the machine by you have no idea who read it, or who didnt. Fax machine are typically MFP devices, so now you may have some part of a medical record on your photocopier HDD.

Many fax machine can be programmed over the wire, so maybe you have default pins and now your faxes are being forwarded and you don't know.

[EamonnMR]

> It's easy to see if the other end successfully received something

This was not the case when I worked in healthcare. It was a constant back and fourth of "did you receive it?" over the phone.

[ses1984]

Some points seem contradictory. How do faxes have history that's easy to see, and if the other end successfully received, but it doesn't have delivery receipt like email?

[keithnz]

not sure I'm understanding? But fax sends data via a protocol, it knows it has sent by virtue of the protocol completing. The fax can keep trying sending and it will keep track of those faxs that have gone through vs those that haven't. Email doesn't have delivery receipts....it's either done by bolting it on in clients or various hacks used by spamme....errr..email marketing systems.

[ses1984]

Just because you get a receipt from the protocol doesn't mean it made it from the fax machine to the intended recipient. Folks who send faxes still regularly follow up with calls and emails, "just sent the fax did you get it?" and the answer is often "no, what fax?"

[kungito]

All these issues could easily be resolved on the internet if someone bothered a bit. Keep a whitelist of connections, private-public key communication (you can exchange keys off internet if needed), receive and read confirmation etc. It's not internet's fault that some idiot is storing passwords in plaintext and/or sending them over unencrypted connections

[TuringNYC]

I see you point in theory, but as a victim of identity theft, from what I can see almost none of these are enforced punishments. From what I experienced and from what I've seen friends experience in the US: - Someone can steal money in your bank account - Sign up for an expensive phone plan and get a $1000 iPhone upfront - Use your healthcare benefits

And you basically file a police report, hardly anyone cares, and you are left with a bunch of paperwork to go fix it yourself. You get the money back eventually with plenty of paperwork.

At which point do thieves committing all sorts of punishable crimes actually get punished? In my case, the person signed up for a line of credit at Lowes, purchased 20k of construction goods, presumable all in a videotaped store and got off scott free.

[pabs3]

Fax machines are just as insecure as that server. Last year taking over a network using just a fax number was demonstrated:

https://research.checkpoint.com/sending-fax-back-to-the-dark...

[roywiggins]

And because it's explicitly grandfathered in to HIPAA as a "secure" method of transmitting patient data.

Also, fax machines are very often just as internet connected as anything else. Email to fax, fax to email, fax-over-IP, it's not just modems dialing each other on copper anymore.

[jahewson]

I’m willing to bet that most digital PBXs out there could be infected by a hacker from halfway across the world too.

[pixl97]

Saw that happen yesterday. A vendor had insecure remote access setup to an older NEC PBX. Someone attacked it and was making international calls with it.

[pretendgeneer]

No Hospitals use fax because it's too hard to change, most information traveling through fax is sent via automated fax servers so it is the worst of both worlds, hack-able server and unencrypted transmission protocol.

[msla]

> Hospitals still use fax because it is a much more punishable crime to tap phone lines which requires physical access

Punishable, sure, but that's CYA thinking. It's less secure, because there's no way to encrypt fax like you can encrypt email. Punishment doesn't help anyone except the CEOs, unless, of course, it was the CEO's information that got leaked.

Also, yes, phone calls are sent over the Internet just like emails. The big difference is, yes, that phone audio isn't encrypted.

[EamonnMR]

Hospitals use fax because A) they think it's secure (it's not) and B) they think it's easier (it isn't)

[DEADBEEFC0FFEE]

That is not why they use them. Doctors like scribbling, and hate being told what to do, that's why.

[AzzieElbab]

it has nothing to do with security. Nurses like fax machines because it gets them a break. I've seen them print e-refferals just to fax to each other.

[HeWhoLurksLate]

That's not even the same thing.

People mess around with just about anything they can get their hands on- so what if the nurses send messages to each other and have fun? I'd be willing to give the people who take care of me a ream of paper if it meant they were in a good mood.

[AzzieElbab]

It has nothing to do with fun or messing around. It is about people being incentivized to avoid the tooling. I worked in US and Canadian health IT, my perspective is completely different from patient's. This may not have huge impact in high-end American hospitals, but in Canada where hospitals are underfunded and drowning in beuracracy it is a disaster

[EuphoricEmu]

Could not agree more! As someone who works as a contractor for two relatively large RHIOs I was absolutely dumbfounded by how contradictory everything is when I first started. I’m in a unique position where I get to interact with dozens of EMR/EHR vendors. They’re all terrible.

[qrbLPHiKpiux]

Cerner? Epic? Suck and suck!

[gesman]

I involved in building 2 systems for healthcare:

Patient privacy monitoring (ensure no one peeks into patient records inappropriately) and medication analytics platform (monitors suspicious and anomalous activity related to drugs and opioids diversion).

This of course deals with multitude of different systems, EMR's, data formats, legacy-everything etc, but the GREAT about this whole thing - is opportunity.

No one really tackled a challenge of visibility across different systems and data formats in Healthcare and it's lots of fun to be part of that.

("Fun" is non-corporate word - but it still fun!)

[specialist]

re: privacy monitoring

I prototyped an access logger based on tamper evident logs, which used rolling hash codes. Precursor to this blockchain mania.

re: visibility

I'm still bullish on the Translucent Database thesis. TLDR: Use the password salt + hash technique to encrypt data at rest at the record level.

--

Source: Designed, implemented, supported 5 regional healthcare exchanges. eg Brooklyn Health Information Exchange (BHIX): https://www.itnonline.com/content/medplus-implement-clinical...

[VectorLock]

Does Epic use MUMPS? I know a lot of professional nurses and the rancor around Epic is off the charts.

[roywiggins]

Backend is all MUMPS. Frontend was for a long time coded in Visual Basic 6.

VB6/MUMPS stack is... not ergonomic to code in.

Epic is easy to hate (it's everywhere), and for good reason. However, the alternatives are not obviously better unless there's been some radical innovation. There are definitely systems designed for a particular piece of a hospital (ex, ER, or labs, etc) that are probably better than Epic is, but when it comes to having one system for the entire hospital, they're all pretty bad.

The main problem is that the customer is not the nurses, it's the legal/financial/administrative side.

[burnte]

> The main problem is that the customer is not the nurses, it's the legal/financial/administrative side.

This. The reason my medical staff like me is that I fight for usability for them, and push back against legal when they make requests that aren't backed up by the regulations. Legal hates me for the same reason, I know the regs and I'm willing to fight them on it.

[crankylinuxuser]

Happen to have an email I could send you? It's sigint security stuff you might want to look into.

My email is jwconway at protonmail dot com.

[JBlue42]

>Epic is easy to hate (it's everywhere), and for good reason. However, the alternatives are not obviously better unless there's been some radical innovation. There are definitely systems designed for a particular piece of a hospital (ex, ER, or labs, etc) that are probably better than Epic is, but when it comes to having one system for the entire hospital, they're all pretty bad.

Yeah, our org is on Cerner and there's excited talk around going to Epic in the future. People don't seem to understand that it's just going to be more of same in dealing with an old, monolithic system.

[btown]

Did anyone think to make languages that compile to those? I imagine that it would make an incredible amount of sense for Epic, at its scale, to write the equivalent of Typescript to reduce errors and improve productivity. Or do they just not have any sort of dev tools/research department?

[roywiggins]

They built their MUMPS IDE themselves, and their change tracking / bug management system. And effectively the database code is also entirely written by Epic- the business logic and database are both written in MUMPS and live side-by-side.

So they have quite a lot on their plate when it comes to tooling. When I was there, there was a truly horrible attempt at a DSL for code-generation that would build database querying code, but I found it both unwritable and unreadable.

[rmtech]

> VB6/MUMPS stack

That sounds like a medical condition.

And after looking into the MUMPS language, I can see why they named it after a disease!

[Forge36]

It does. The language predates no-sql though it hasn't been kept as up to date as C. It's a database language.

[Sniffnoy]

Epic uses MUMPS, but a nurse would not typically be interacting with that side of it...

[Forge36]

As it should be. Why should companies brag about using SQL or Hadoop?

[snuxoll]

Yes, as does Meditech. Both are horrible products to integrate with and they’re rather common in large hospital systems.

[dpflan]

Regarding legacy EMRs, are specifications/standards like HL7's FHIR actually gaining any traction and making data interoperability more feasible?

[sixothree]

In my world they sure are. Want health records on your iPhone? Well that comes via FHIR. You can even see the FHIR resource JSON in the Health app.

But there are many systems in a hospital. And as EuphoricEmu pointed out within the hospital, admits, discharges and movements throughout the hospital are still done via HL7v2 (a delimited and structured format).

Additionally, I would absolutely NOT build a new system on HL7v2 at this point in time. I would only use it to integrate with existing systems.

Also, I do know of EHR systems that use FHIR for their internal data storage format.

[EuphoricEmu]

Yup, there are a few EHR vendors that have some limited support. Intersystems’ whole platform is shifting towards have pure native support for FHIR. It’s still very rough and to into the weeds of things you sadly have to delve into that very old language (objectscript = MUMPS). But thankfully there bindings for more modern languages for it.

[snuxoll]

Unfortunately consumer-facing use cases are the ones big EMR vendors like Epic are focusing on with FHIR, push is often neglected or left out meaning we still have to rely on good old HL7v2 to get real time feeds out of the system.

[EuphoricEmu]

Yes exactly! Since patient movement within a hospital is done primary with v2 and then hours or even days later a patient summary CCD might get sent across. Unless you’re dealing with encounter based CCDs. But those still aren’t live.

[sixothree]

FHIR Subscriptions do exist.

[EuphoricEmu]

Not OP but I can tell you that due to the object size limitations of FHIR objects, document exchange is just not feasible. Using FHIR to register a patient is pretty pointless currently as every older and larger hospital sends HL7v2.

RHIOs building gateways for 3rd party apps is very much the future of FHIR. But they’ll still be interacting with that crufty legacy system.

Just my own two cents.

[dpflan]

Thanks! I was not aware of the object size limitations nor of RHIOs

Are RHIOs using engineers/consultants to wrapped legacy systems then devise methods to make accessing that data easier -- how are RHIOs using FHIR and legacy system to construct the future of medical data interoperability?

[EuphoricEmu]

RHIOs exist predominantly to allow the exchange of patient information between participating hospitals or even regions. I personally only have insight into how two RHIOs operate. Both are very interested how they can turn FHIR into another avenue to distribute patient information.

The issue is that most EHR/EMR vendors have very limited limited interoperability with FHIR. As a lot of these vendors are struggling with CCD 2.1 implementations. The exciting space is allowing patients through 3rd party apps to request specific information or even send it to the HIE (RHIO in this case) and let there GPs know of certain events.

Which means a lot of work to bend over backwards to get APIs exposed for these 3rd parties.

[lmiller1990]

I work as a company that builds software to optimize dosing, and we have apps in a few of the major EHR vendors app stores. FHIR has been great for getting and presenting data - although I think true interop between EHRs using FHIR would be challenging, due to the large amounts of data and the (relative) complexity of each EHR. Despite everyone using FHIR, it does suffer from the same problem other specs like HTML do, everyone has an ever so slightly different implementation, and once it's been implemented, no interest in changing or improving it.

[wdb]

There are new data exchange protocols/formats which help to exchange between parties. They have been defined 2000+ so they are quite modern.

The only problem is that is such a diverse way the healthcare providers are implementing it. So you end up having provider specific code :)

[specialist]

True.

My ETL work in healthcare finally pushed me to treat the problem as screen scrapping. Bypassing all the attempts at formality, eg XSD.

The next step, which I prototyped but was put into purgatory by being acquired, was to simply capture all the data and use text retrieval tools, eg Lucene. Versus ingesting the data, normalizing it (to some schema) and populating database(s). Basically, postponing the translation/transformation of client data until viewing. Because, as you hinted, everyone does stuff differently, and clients generally have no idea what their data looks like until we showed them.

The proper solution would be to have direct access to source data, versus data feeds, but that ain't gonna happen until we have single payer, because the current incentive structure strongly discourages such simplicity.

[DEADBEEFC0FFEE]

So true. Also healthcare software companies are shipping shit quality products and charging a fortune. Hospital CEOs are buying from mates. Fancy pants doctors are demanding exceptions. Shift staff are sharing passwords. Medical systems are not even capable of automating proper user access controls. The volume of data, especially medical imaging is growing at a crazy rate. Network links are under invested in. They sweat the assets to the points of lunacy. Ok I'm gonna stop.

[JBlue42]

Pretty much sums up my experience so far in healthcare IT, esp. the C-level and doctor part.

[jcims]

Don’t forget the turnkey Siemens PET scanner complete with virus infested control PCs.

[Gatsky]

What if there was an open source EMR? *nix backend with browser interface. The huge amount of money saved could pay for support, customisation and implementation. Perhaps I’m being naive, but a system where the institution has more ownership would go down well I think. We are about to get an Epic EMR and all the clinical staff are basically expecting a catastrophe.

[wslack]

The VA's old system (VistA) is that open source solution - but other hospitals didn't want to use or buy it.

I think the EMR industry is driven more by safety, liability, and revenue for hospitals (in that order) than by patient/physician desired features or security.

It's also difficult to build tooling that interconnects across all of the medical specialties - and with the amount of customization that some providers want.

[0point692]

You could look at this from a couple of angles. From the healthcare business side, it's controlling costs, plus either regulatory compliance or revenue support (e.g., through provider lock-in). From the tech side, it's whether something developed 'in house' can be 'sold' to the private market: the economics about government institutions 'crowding out' private ventures argues that publicly funded innovations should not deny private profit opportunities. In the public sector this is controlled with ethics regulations that prevent organizations (and individuals) from benefiting from that effort: they legally cannot market or sell that work _at all_. With an open-source system, it's (business) risk management and (provider) lock-in concerns. Proprietary systems are used to satisfy both. Safety risks arising from cybersecurity concerns are almost always chained to something already managed by existing processes with an insurer at the end of the line.

[Gatsky]

Hey thanks, I hadn't heard about it [1]. Apparently VistA has the highest user satisfaction of any EHR! Sounds like what is missing is the equivalent of Red Hat, which can support and implement the product, although implementation would be much more involved.

[1] https://en.wikipedia.org/wiki/VistA

[ikeboy]

Is VistA usable by sole practitioners?

[TeMPOraL]

Like GNU Health?

- https://www.gnuhealth.org/#/download/projects

- https://savannah.gnu.org/projects/health

- https://en.wikibooks.org/wiki/GNU_Health

Can't tell if it's any good, though I hear it's been used in some medical facilities around the world.

[bsder]

> Healthcare is VERY backwards from a tech standpoint.

This strikes me as the SAP problem.

Everybody hated their big company's <foo> system. Until SAP came along and somehow made it orders of magnitude worse.

It's what happens when purchasing decisions are driven from the top. Sadly, in healthcare, given all the regulations, I'm not sure there is another option.

[agumonkey]

How much end to end efficiency do you think a proper/average IT healthcare system would bring ?

[ethbro]

The amount of complexity that it would be necessary to simplify and approximate would make any answer to this question meaningless.

And it's not only an IT systems problem. It's a comprehensive systems problem.

Which includes training, and counterparty expectations, and manual data entry, etc.

[agumonkey]

I'd wish hard to have a peek in these projects.

[ethbro]

One of the reasons change is so slow in the industry is because there are many must-be-coordinated changes, with various independent parties.

E.g. if I update my system, you need to update your system

From what I saw, one of the biggest motivators would be carving out legal protections for trialing some smaller % of total workflow under new systems.

E.g. If you moved < 5% of your claims handling to a new automated system, and it started rejecting claims that there was an informal understanding would be patched up on the insurance side, then nobody could be sued via intra-party contracts (but still beholden to national / federal guidelines, of course)

By decreasing that first burden of migration, you might get more traction in aggressive IT updates.

[Scoundreller]

> E.g. if I update my system, you need to update your system

I see you’ve played InternetExplorer X vs X+1 before.

[burnte]

> How much end to end efficiency do you think a proper/average IT healthcare system would bring ?

I have a long history in proper IT, and I'm very legally/regulationally knowledgeable, and in my two healthcare gigs I've made friends of the medical staff for improving responsiveness in IT and making things easier to use, while also reducing security problems by having both of those worlds of knowledge. Usually top IT management isn't technically knowledgeable, and frequently they're not even that good with regulatory knowledge. That makes it hard on the rank and file to be efficient. Not to pay my own back too much, but being well versed in both regs and tech helps a LOT in user satisfaction.

[Scoundreller]

What were you able to make easier to use?

[delfinom]

>we use very, very few off the shelf solutions.

Even off the shelf is no solution really. Everything is proprietary, you get locked in and years later the vendor isn't maintaining shit and moving away becomes an blackhole of a cost sink.

>It's INCREDIBLY common for nurses and doctors to need to have administrator access on their Windows desktops for various apps.

Honestly, needing admin on a "user network" device isn't the worst. You can still run malware that attacks the network via non-admin context :/ The best move is to use AppLocker if it's critical.

[celdon25]

> Don't like the terminal interface? Let's slap a GUI on the front that still interacts via TTY on the back end.

This made me smile, because this is exactly what I did when working for a major health insurer...

[goatinaboat]

Let's build it in MUMPS, a 51 year old language that originated on the PDP7

It’s funny. You got Ruby guys out here who think every problem can be solved with a new DSL, and you guys actually have a DSL and want to get rid of it! Maybe the grass is always greener on the other side?

Nah. C

Also traces its roots to the PDP7 https://en.wikipedia.org/wiki/C_(programming_language)#Early...

[Forge36]

I've never understood hating on mumps. 66 vs 72 (c language) It's just shitting on the language for being old without a way to impact it.

[burnte]

It's so much more niche than something more widely used like C that it makes me very suspect.

[GarrisonPrime]

From my perpective, part of the problem is that any system is chosen based on political connections and/or bribes. And once any system is in place, there is zero reason to replace it until someone else puts down enough bribe money to convince a poltician to make another move.

Government promises, and thus government kills. A monopoly enforced by guns and prisons.

[specialist]

Despite the awesomeness of our tech stack and execution, our customers (hospitals) were completely fleeced by our sales and project management teams.

During the mid aughts, hospitals simply didn't have the experience to defend against predatory consultancies.

Hopefully that situation has improved with the addition of people like you.

[dgzl]

Sounds like local and state government.