[Gatsky]

Well you discount the most important thing in the first line. 'Cutting-edge' is a funny way of saying 'most effective', as if it were somehow irrelevant.

I didn't say most evidence is from phase III RCTs, particularly if you include everything that happens in oncology as the denominator, only that meta-analyses were not that relevant. Most of the critical patient facing interventions have the backing of good quality trials, at least where it is reasonable and possible to do a trial. Also one of your citations is seemingly casting doubt on the value of meta-analyses in oncology, so somewhat confused about your point.

That paragraph from NCCN is quite interesting. It is describing medicine in general really, and belies the fact that oncology has probably one of the strongest evidence base across all medical fields. Take for example how many stents cardiologists have inserted long after contradictory evidence was available, or how many pointless back operations have been done, or how many people have sat through fruitless psychoanalysis.

[haldujai]

> Well you discount the most important thing in the first line. 'Cutting-edge' is a funny way of saying 'most effective', as if it were somehow irrelevant.

I'm discounting it for this discussion because your argument is:

"meta-analyses are somewhat irrelevant" and "Meta-analyses are mostly there for trainees to notch up a paper." which is completely false.

Note a single clinical trial is still only considered "good quality" while multiple trials or meta-analyses are considered "high quality".

To address this new point you raised, when something has very promising early results we start using it in treatment (e.g. 3rd gen TKIs in adjuvant NSCLC) but until this weekend we had no 5 year OS survival for adjuvant use.

It's entirely possible something one thinks is "most effective" is later proven to not be (gen 1-2 TKIs, HIPEC, etc).

> That paragraph from NCCN is quite interesting. It is describing medicine in general really, and belies the fact that oncology has probably one of the strongest evidence base across all medical fields.

> Take for example how many stents cardiologists have inserted long after contradictory evidence was available, or how many pointless back operations have been done, or how many people have sat through fruitless psychoanalysis.

I'm not sure what point you are trying to make by addressing other specialties.

The National Comprehensive Cancer Network, comprised of multidisciplinary experts from 33 of the leading cancer centers in the country, is unequivocally the authority in oncology and is incredibly well respected. I'm going to defer to their opinion on the quality of evidence available and the hierarchy of evidence.

> Also one of your citations is seemingly casting doubt on the value of meta-analyses in oncology, so somewhat confused about your point.

The JAMA article states that the methodology in many studies does not meet NCCN/PRISMA criteria which is a well known, this says nothing about the relative value of good-quality meta-analyses (which are far more common now with the PRISMA update).

I'm really not sure why you think systematic reviews are irrelevant, this is a very radical viewpoint that I've seen no evidence of. Good meta-analysis > good RCT. The reality is that good quality studies of both types are uncommon in medicine, but the goal is still to use good SRs.

[Gatsky]

I don't think it is false. I can only speak of my experience as an oncology healthcare provider. I spend many hours each week digesting the literature, and <5% of that involves meta-analyses. In the multidisciplinary meetings I chair, we rarely discuss evidence from meta-analyses, but we are always talking about clinical trials. The NCCN guidelines were useful when I was a trainee, but otherwise they are too US-centric, and they are always out of date due to the frequency they are updated. This is why ASCO keeps issuing rapid updates in breast cancer for example (https://old-prod.asco.org/practice-patients/guidelines/breas...). There are 2 such updates this year already. If the primacy of meta-analyses were so great, why would they bother to issue rapid updates of what you class as low quality evidence?

But to give a concrete example, the problem with meta-analyses is well illustrated in the recent EBCTG meta-analysis published in the Lancet, a top tier journal. This involved over 100,000 patients, and explored concurrent chemotherapy regimens in breast cancer. The problem is that such regimens are not used anymore. The authors acknowledge in their own conclusion that this massive meta-analysis contradicts their own previous meta-analysis showing the superiority of sequential therapy. What exactly does one do with this? How does this help a patient get the right therapy? The treatment of various breast cancer subtypes has also evolved so much that the trials they meta-analyse are mostly obsolete. Hence my point, that meta-analyses are just not that useful in oncology, even truly massive well conducted ones published in prestigious journals. So it is not so simple as meta-analysis > RCT, that is merely lazy dogma. I find it hard to believe that anyone actually treating cancer patients would hold this view.

Of course most meta-analyses in oncology are not 100,000 patient behemoths conducted by consortia. They are much smaller studies, which usually don't bother to get patient level data, and just copy numbers from tables in the original papers while running through the Cochrane systematic review template.

And yet, here I am dubbed 'radical' at the bottom of a comment thread on Hacker News. Unfortunately the dogma around systematic reviews and EBM has exceeded its usefulness by quite some margin. The meta-analytic method was developed by psychologists trying to compile evidence about extra sensory perception of all things - an inauspicious beginning if there ever was one for the supposed cornerstone of medicine.

[haldujai]

Speaking of ASCO, their methodology for recommendations is conducting their own systematic review and they start with Cochrane.

>Upon approval of the Protocol, a systematic review of the medical literature is conducted. ASCO staff use the information entered into the Protocol, including the clinical questions, inclusion/exclusion criteria for qualified studies, search terms/phrases, and range of study dates, to perform the systematic review. Literature searches of selected databases, including The Cochrane Library and Medline (via PubMed) are performed.

>After the systematic review is completed, a GRADE evidence profile and summary of findings table is developed to provide the guideline panels with the information about the body of evidence, judgments about the quality of evidence, statistical results, and certainty of the evidence ratings for each pre-specified included outcome.

Rapid criteria:

The criteria for a rapid recommendation update are: 1. that the identified evidence is of high methodological quality, 2. there is high certainty among experts that results are clinically meaningful to practice, 3. the identified evidence represents a significant shift in clinical practice from a recommendation in an existing ASCO guideline (e.g., change from recommending against the use of a particular therapy to recommending the use of that therapy; or a reversal to a recommendation) such that it should not wait for a scheduled guideline update.

A systematic literature review focused on the updated recommendation will be conducted by ASCO staff. Specifically, the immediate past guideline literature search strategy will be updated and filtered by search criteria specific to evidence informing the recommendation under review. All identified evidence will be quality- appraised using the GRADE methodology as outlined in Section 10 of this ASCO Guideline Methods Manual. The procedures used to draft the rapid recommendation update and deliberations by the expert panel will follow routine methods for all guidance products as outlined in this ASCO Guideline Methods Manual.

ASCO position on meta-analyses:

All these reasons can be used to make the excuse that a systematic review and meta -analysis should not be done, especially if resources aren’t available to hand search all journals of all languages, etc.

The solution is not to avoid doing a systematic review or meta-analysis, but to reveal to the reader what short cuts were taken (e.g., we included only peer reviewed published studies, or restricted our eligibility to studies published in English). This shows transparency, and then the readers can decide how important this problem is in applying the results of your meta -analysis to their situation.

Myths about meta-analyses • A literature-based meta-analysis is not worth doing

So systematic reviews and meta-analyses are no longer useful?

That you're arguing with anecdotes is proof itself of why EBM is important.

>There are 2 such updates this year already. If the primacy of meta-analyses were so great, why would they bother to issue rapid updates of what you class as low quality evidence?

Both March 2023 updates are identical.

https://old-prod.asco.org/sites/new-www.asco.org/files/conte...

https://old-prod.asco.org/sites/new-www.asco.org/files/conte...

https://old-prod.asco.org/sites/new-www.asco.org/files/conte...

[Gatsky]

Read the ESR1 rapid update. The methods say:

"A targeted electronic literature search was conducted to identify any additional phase III randomized controlled trials in this patient population. No additional randomized controlled trials were identified. The original guideline Expert Panels reconvened to review evidence from EMERALD and to review and approve the revised recommendations."

Where is the meta-analysis? Where is the funnel plot? What are you even arguing about? They issued an update because of one trial.

Here is another one from June 2022, a major change to how one type of breast cancer is managed, in the methods:

"A targeted electronic literature search was conducted to identify phase III clinical trials pertaining to the recommendation on immune checkpoint inhibitors in this patient population. No additional randomized trials were identified. The original Expert Panel was reconvened to review the key evidence from KEYNOTE-522 and to review and approve the revision to the recommendation."

Where is the meta-analysis? Again, what are you trying to argue? They issued an update because of one trial.

There are two updates this year, one about HER2 testing, and one about ESR1.

[haldujai]

Perhaps you’re unaware but not every systematic review is or can be a meta-analysis. Meta-analyses can and are also included in systematic reviews released by ASCO.

The criteria for a rapid update is listed in the comment you replied to, I have no misunderstanding of why they publish them but you are trying to misrepresent this as deviating from EBM.

Your words were: “Unfortunately the dogma around systematic reviews and EBM has exceeded its usefulness by quite some margin.”

I’m not sure what point you’re trying to make but seeing as you’re attempting to argue ASCO’s own position and methodology with anecdotes and conjecture based on one specific area of breast cancer treatment and extrapolating to the entire field of oncology I’m not sure there’s a point in engaging further.

You can refer to the full ASCO statement I linked which discusses meta-analyses for their arguments.

Cheers.