[patel011393]

This may seem like bad news, but as a grad student studying evidence synthesis I can say that there is a silver lining here. The quality of studies in systematic inputs is low and even Cochrane's reviews have been critiqued for missing things. It's a big factory that hasn't fundamentally innovated how research is synthesized. There are newer forms of synthesis like realist reviews, meta-modeling, and model-driven meta-analyses (those are just a few examples).

Hopefully this results in some innovation at the conceptual and financial levels. For those who lost their income, they have transferrable skills that would be welcomed in healthtech companies, other governmental groups, and academic positions. It's really not as bad as it seems.

[haldujai]

> The quality of studies in systematic inputs is low and even Cochrane's reviews have been critiqued for missing things.

Criticism is good, expecting perfection is not.

Quality issues are a problem of the underlying research rather than the review process. High quality reviews, like Cochrane, assess risk of bias which can limit the strength of conclusion they make where the underlying studies are poor quality.

> There are newer forms of synthesis like realist reviews, meta-modeling, and model-driven meta-analyses (those are just a few examples).

All of these answer different questions/have different use cases than a SR and are much harder to do while maintaining comparable quality and risk of bias.

What's your gripe with high quality SRs following standards like PRISMA, STARD and QUADAS-2?

[TheBlight]

I don't remember hearing anything about Cochrane in mainstream news until they put out an unfavorable review on the effectiveness of mask usage during covid 19.

[ggm]

A very unfortunate, badly run meta analysis as I understand it which fundamentally undermined trust in their governance because it was so easily critiqued. I haven't seen a robust defence of that one.

[TheBlight]

I haven't seen compelling critiques of it[0]. Curious to read them.

Their findings:

On medical/surgical vs. nothing:

"Wearing masks in the community probably makes little or no difference to the outcome of laboratory‐confirmed influenza/SARS‐CoV‐2 compared to not wearing masks (RR 1.01, 95% CI 0.72 to 1.42; 6 trials, 13,919 participants; moderate‐certainty evidence)"

On N95/P2 respirators vs. medical/surgical:

"The use of a N95/P2 respirators compared to medical/surgical masks probably makes little or no difference for the objective and more precise outcome of laboratory‐confirmed influenza infection (RR 1.10, 95% CI 0.90 to 1.34; 5 trials, 8407 participants; moderate‐certainty evidence)."

[0] https://www.cochranelibrary.com/web/cochrane/content?templat...

[ggm]

Thats a revision of a paper with roots back to 2006. I applaud them for having a rich source, but the primary objections can be understood to be in their apologia:

https://www.cochrane.org/news/statement-physical-interventio...

The original Plain Language Summary for this review stated that 'We are uncertain whether wearing masks or N95/P2 respirators helps to slow the spread of respiratory viruses based on the studies we assessed.' This wording was open to misinterpretation, for which we apologize. While scientific evidence is never immune to misinterpretation, we take responsibility for not making the wording clearer from the outset. We are engaging with the review authors with the aim of updating the Plain Language Summary and abstract to make clear that the review looked at whether interventions to promote mask wearing help to slow the spread of respiratory viruses.

The problem was how they communicated. The data is the data.

[csours]

My opinion (feel free to have another one) is that it points to the need for training and more comfortable masks. The studies weren't WRONG per se, but they are very hard to interpret.

My whacky opinion: human facing sciences (medicine, psychology, diet, sociology, economics, etc.) have a huge problem with solutions that require motivation to succeed. In other words, if you are motivated, you can make masks work. If you are motivated, you can make intermittent fasting work. Same for EVs, Solar Panels, etc.

[pella]

[EBM analysis]

"Adapt or die: how the pandemic made the shift from EBM to EBM+ more urgent"

https://ebm.bmj.com/content/27/5/253

"Evidence-based medicine (EBM’s) traditional methods, especially randomised controlled trials (RCTs) and meta-analyses, along with risk-of-bias tools and checklists, have contributed significantly to the science of COVID-19. But these methods and tools were designed primarily to answer simple, focused questions in a stable context where yesterday’s research can be mapped more or less unproblematically onto today’s clinical and policy questions. They have significant limitations when extended to complex questions about a novel pathogen causing chaos across multiple sectors in a fast-changing global context. Non-pharmaceutical interventions which combine material artefacts, human behaviour, organisational directives, occupational health and safety, and the built environment are a case in point: EBM’s experimental, intervention-focused, checklist-driven, effect-size-oriented and deductive approach has sometimes confused rather than informed debate. While RCTs are important, exclusion of other study designs and evidence sources has been particularly problematic in a context where rapid decision making is needed in order to save lives and protect health. It is time to bring in a wider range of evidence and a more pluralist approach to defining what counts as ‘high-quality’ evidence. We introduce some conceptual tools and quality frameworks from various fields involving what is known as mechanistic research, including complexity science, engineering and the social sciences. We propose that the tools and frameworks of mechanistic evidence, sometimes known as ‘EBM+’ when combined with traditional EBM, might be used to develop and evaluate the interdisciplinary evidence base needed to take us out of this protracted pandemic. Further articles in this series will apply pluralistic methods to specific research questions."

[haldujai]

I don't understand, this paper is describing how EBM works today and calling it EBM+ for some inexplicable reason. This is a long winded rant basically concluding with face validity is important...

[cyrillite]

This is an insane ask, but I would love to hear more about various reviewing and synthesis styles, their strengths and weaknesses, which are particularly promising and why, etc. you’re welcome to link me to material, but I’d also appreciate a brief overview.

I have nothing to offer other than thinking that this is a super cool area of study and I’d love to hear more

[thedailymail]

This site provides a concise description of many review types: https://guides.mclibrary.duke.edu/sysreview/types

[ttpphd]

Totally agree. Cochrane is a dinosaur and can do more harm than good.