| I read the original study. I have a few thoughts. My partner is a physician and I am an AI researcher working in medicine. I think a lot about doctors as machine learning models, and RCT results like loss terms in a complicated objective function. What is the best learning rate for updating physicians (our models) from the results of RCT's (part of our loss)? The authors reviewed all articles in three journals from (generally) between 2003-2017. They didn't, afaict (please point me if they did), review the time-to-correction (if any correction has been made). It takes some time before the results of an RCT end up in established practice. I'm actually surprised it's so small in many cases. It's not like there's a database where the results of every RCT are immediately updated and the physician model is retrained overnight on the new data. Even if there were, imagine if the learning rate (so to speak) were so high that every discipline immediately changed their published best practices on the basis of a single RCT? Here's a cautionary paragraph from one of the excellent reversal studies they use: Several limitations of the study warrant discussion. First, because we enrolled only 26% of eligible patients, our findings must be generalized cautiously. The most frequent reason that patients declined enrollment was a strong preference for one treatment or the other. Since patients' preferences may be associated with treatment outcome, our trial may be vulnerable to selection bias. Participating surgeons may not have referred potentially eligible patients because they were uncomfortable randomly assigning these patients to treatment; this form of selective enrollment may also create bias.26 Second, because the trial was conducted in academic referral centers, the findings should be generalized carefully to community settings. Third, we did not formally assess the fidelity of the physical therapists or surgeons to the standard intervention protocols. Finally, our study was not blinded, since our investigative group did not consider a sham comparison group feasible. [0] I'm less concerned about RCT to Best Practice time than from Best Practice to Typical Physician Practice time. There is a cascaded model connected to the 'complex RCT loss' and it's discipline published practice down to individual physician treating patients. Compressing the time from RCT to individual physician is fraught with difficulties, but could be improved. Finally, RCT is the gold standard, but it's not perfect and it doesn't always clearly translate to the individual physician's model of practice. Many best practices weren't established from RCT's either. And an inconclusive result from an RCT is not the same thing as proving that there's no difference in outcomes, but a proper statistician can chime in there. [0]https://www.nejm.org/doi/full/10.1056/NEJMoa1301408 |