[PiggySpeed]

No, you don't understand this.

The Lancet is one of the most impactful journals in medicine, and has played an important role in shaping how medicine is practiced. It is rigorous in its process, but it's not foolproof. So yes, there are some bad studies that come through, but that is why healthcare professionals are TRAINED to interpret studies--we separate all the bullshit from the legitimately useful stuff.

You should see some of the crap that comes through in some of the lower-impact journals.

> The combination of preprint servers + twitter has proven far more effective in looking after humanity’s best interests.

This view is incredibly false and dangerous for so many reasons.

[loulouxiv]

Well if healthcare professionals are trained to interpret study, and separate the wrong from the useful stuff I guess it won't be so hard for The Lancet to hire some of them to review papers... I tend to believe Didier Raoult when he says in his last interview that anybody who works in medecine could easily tell this paper smelled really fishy, and I am sure it would be less costly for The Lancet than the reputation loss they are going to suffer.

[devin]

You act as if highly-trained professionals don't ever disagree.

[notahacker]

Most professionals found Raoult's own published claims of the 100% cure rate of hydrochloroquine combined with azithromycin based on small groups without proper controls fishy, for a start.

[aksss]

I tend to agree with you in whole, with caveat that past contributions don't speak to current value, necessarily. I'm always reminded of the ruthless statement, "God cares not what you have done, but what you are doing". I say it tongue-in-cheek, but there's wisdom to it.

Also, can we all stop describing any view or information as "dangerous"? It sounds like my mom, or religious people from the 80's.

False is one judgement and objectively proven.

"Dangerous" smacks of a censorious nature, which is subjective and, sorry, often based on a conflict of interest/agenda.

Encourage everyone to read Matt Taibbi's recent article about the recent rise of "virtuous" censorship as food for thought: https://taibbi.substack.com/p/temporary-coronavirus-censorsh...

[bobcostas55]

You are confusing impact with quality. They are not the same thing.

[macinjosh]

> So yes, there are some bad studies that come through, but that is why healthcare professionals are TRAINED to interpret studies

So healthcare professionals are trained to separate the wheat from the chaff, but peer reviewers are not?

[ColanR]

> You should see some of the crap that comes through in some of the lower-impact journals.

Thank goodness that "healthcare professionals are TRAINED to interpret studies."

[Gatsky]

I am a healthcare professional, and I've published in one of the Lancet Journals. I understand perfectly fine.

Here's the thing - The Lancet is not impactful. Researchers choose to publish their impactful research in The Lancet. Other researchers donate their time to peer review this research. It is the researchers that have shaped how medicine is practiced, not The Lancet. It is the researchers and the peer reviewers that are rigorous. The Lancet itself is none of these things, it is a business run by people who don't do research. Without the people who actually do the work, or the patients that volunteer for the research, it is nothing. The Lancet, like all top tier journals, has long forgotten this distinction.

The problem isn't healthcare professionals lacking training. Take the Wakefield study. Did any doctor decide to stop offering vaccines because of that study? No. But it had a large impact on a the anti-vax narrative. These top journals have influence far beyond the professional sphere. This is why your suggestion we should give them a free pass is dangerous.

I think it is clearly true that rapid publication and out in the open discussion and peer review is very healthy. I don't see how you could argue otherwise? Why wouldn't I want to be able to read the opinion of someone I respect intellectually eg Andrew Gelman, on a study they have decided to comment on? How is that 'dangerous' if I am a healthcare professional who should be able to critically review a published paper as you suggest? Why do I need to rely on the reviewers the journal has chosen? The Journals want to keep things as they are to maintain their importance and their bottom line. We suffer as a result.

[rscho]

Healthcare pro also. You are right. Academic publishing has become extremely toxic to both science and the healthcare system in general.

A major issue with this is the political and hierarchical benefits of publishing. It encourages professionals to focus on that, to the point that clinical work is now looked down upon and producing loads of shitty papers will propell you to the forefront of the academic star system and make you rich. It's truly cancer for our healthcare system.

[cm2187]

But I keep hearing that the definition of good science is "peer reviewed" science. So you are right that it is not the peer review that makes the science, but what about the science us laypeople can rely on? How else can we tell a serious claim from a fantasist one?

[skeletor_999]

The scientific method is hard to explain, and scientists frequently mention peer reviewed papers when talking to the public. I think that this is mainly to dismiss things like health anecdotes that spread around the internet. We don't have time to explain in detail why your uncle's implausible and untested miracle cure for whatever disease is unlikely to work.

There's no easy answer to which sources are reliable, but it helps to look at what reputable organizations say. They typically base their advice on the majority opinions of relevant experts. Scientists are human, so they are bound to make mistakes, but in the whole they have made demonstrable progress in a variety of fields.

[frenchyatwork]

Peer reviewing is mostly orthogonal to journal publishing. Publishing in a journal generally requires a peer review, but there's no reason why you need a journal publication to get a peer review. Journals largely exist to manage prestige and career advancement in academia.

As for what science lay people can rely on? I'm not sure there's a simple answer to that question.

[Natsu]

I think you either have to find scientists you can trust (which presents a chicken & egg process) or study the field yourself.

That said, there are some good heuristics. Good scientists are able to show their reasoning process and explain why X is bad/good science. Bad ones handwave everything and push credentialism or rely on fallacious reasoning.

For example, I remember long ago when Ars Technica posted a detailed explanation of both what homeopathy is and how we know that it's completely bogus, why "water memory" doesn't and cannot exist, etc.

Meanwhile, in another failure of peer review, Nature published nonsense on "water memory" back in 1988:

https://en.wikipedia.org/wiki/Water_memory

[DiogenesKynikos]

How can you criticize The Lancet for a small number of faked studies that have gotten through, but then praise pre-print servers, which are rife with complete nonsense?

Pre-print servers are great for getting results out quickly, but the level of quality control is extremely low. Journals are slower, but have a higher level of quality control.

There is a trade-off between speed and quality control, and both pre-print servers and journals sit at useful points in that trade-off.

Given the pressure to get information that might help fight CoVID-19 out quickly, journals have probably shifted towards the "quick/low-quality" end of the spectrum in the past few months. They'll move back when things calm down.

[rscho]

Journals have been hype/politically-driven since as long as one can remember. And peer-review quality is a joke.

Preprints are not perfect, but I'd wager that anything free and truly open to review is better than what we have now.

[Gatsky]

Specifically I am praising pre-print servers in the current crisis, not in general. I don't propose they can replace journals wholesale in the near future.

The harm done by bad research in the Lancet is colossal because the results are widely read and disseminated, that is obvious isn't it? Pre-print servers are still quite obscure. Anyway, arxiv has been running for a long time with much success in certain fields, and hasn't been overwhelmed with nonsense, so your characterisation of pre-print servers is poorly calibrated and hyperbolic.

Also the Lancet charges money for its products, and can therefore expect to be held to greater account.

[DiogenesKynikos]

So the pre-print servers are better because not a lot of people read pre-prints?

> arxiv has been running for a long time with much success in certain fields, and hasn't been overwhelmed with nonsense, your characterisation of pre-print servers is poorly calibrated and hyperbolic

I just said that the level of quality control on pre-print servers is extremely low - which it is. They check for very little beyond blatant plagiarism and obvious junk (determined within minutes).

I didn't say that pre-print servers are useless. They're very useful, but it's undeniable that prestigious journals apply an additional filter, which is much more exacting.

If you were to pick a random study from a pre-print server, and a random study from the Lancet, which do you think is more likely to be reliable? Which do you think has undergone more thorough peer-review?

[Gatsky]

You don't seem to be reading my comments. I agree with you that pre-print servers are not useless, and that prestigious journals are also not useless. Pre-print servers obviously have a lot of questionable content, this is immediately obvious. As I said, they are not a replacement for peer-reviewed journals (except maybe in the early stages of a pandemic, which was my long departed original point). It is also true that good research gets published in prestigious journals. Lots of good research also doesn't get published in prestigious journals because the editor of the Lancet decides it isn't interesting enough, or they chose clueless or callous reviewers who kill the paper in peer-review, or they just accepted a paper with the opposite result last week or for many other arbitrary reasons unrelated to the quality of the research or utility to humanity.

[creaghpatr]

That they have that level of impact would appear to be problematic, given this news, rather than the other way around.