|
|
|
|
|
by technobabbler
1547 days ago
|
|
The article said that after this incident the manufacturers modified the search to match at least 4 letters, not just 2. And matching the 2 required a special override mode to begin with, but apparently that was a common thing to do. I don't know how these systems are designed and tested, and what regulatory hurdles they have to pass, but it sounds like there is a huge disconnect between how the manufacturer expects them to be used and how they are actually used, with frequent overrides, day to day. It must be a tough industry to work in, either in the patient facing side or the medical devices and software side. I'd hate to be the person who coded all the warnings in that software. "What do you mean they bypassed all 7 warnings? Even the one that said this was a paralyzing agent?!" I wonder if some of this wasn't a procedural failure too. Like why doesn't a potentially life ending drug require at least two people to vouch? Even in retail a manager has to come and turn a key for some trivial refund, or in our field a reviewer has to approve changes first. Is it that the hospital cheaped out on staffing so they didn't want two nurses double-checking each other? Was this use case never accounted for in software development? Are all drugs potentially life ending so there's no way for the software to reliably reduce false positives? So many questions... |
|
There is a certain level of cognitive offloading happening with the machine that is uncomfortable to me. Its like the Tesla autopilot giving drivers a false sense of security. One could easily fall to assuming what came out of the cart is the right thing. People are much less trusting with each other.