[cptaj]

Be careful with this default position.

If we start from first principles: - Any one individual is not trained or experienced in the vast majority of scientific disciplines, regardless of his/her expertise in any one field - Even if half the studies published have flaws (where did you get that stat? was that study flawed?), their body of references is filtered by decades of replication and paper ranking of importance - The people doing the studies have decades of study in the particular field they're publishing about

Concluding from first principles: Its more likely that a random individual's opinion about a random niche of scientific study will be wrong than that of an individual trained on that niche

And then also starting from "received wisdom": Joe Schmuck on the street doesn't know more than scientists about their thing

[cynicalpeace]

The 3 things I mentioned counterbalance each other. And also notice I didn't say we should get rid of science or scientists. I said we have gone too far in the other direction of disregarding "Joe Schmuck" (or granny), which your comment demonstrates.

Also there are lots of Joe Schmucks that know better than a so called "nutritionist" that a steak is better than an Oreo. Not all Joe Schmucks, but a lot of them.

[pclmulqdq]

Reading many scientific studies that are done poorly, I am not convinced that you need a lot of formal training in a field to identify bad science. Many frauds and errors are blatantly visible in their data, which should have a given distribution but doesn't, or in their flawed analyses. Many others have several clearly uncontrolled variables (hint: if it's not in the paper, they didn't control it).

People of a given field will also often circle the wagons around bad actors to protect their own jobs and reputations. This introduces a very strong bias in the people who should be the most highly-informed on their own subject, and sadly makes them less trustworthy than people like Data Colada (https://datacolada.org/) who go beyond their own fields.

[cjbgkagh]

I did stats and applied research so I basically got to read papers as my full time job, then go home and read papers as a hobby.

Most fields are really bad at stats which is one thing that is pervasive across just about all science. More than half of science wouldn’t exist without some sort of fraud, clear selection bias, or p-hacking.

It’s to the point that one sure way to lose friends and be shunned in academia is to be good at statistics.

It’s like the plagiarism scandal, academics have been getting away with it for so long it’s become normalized. It used to be that plagiarism would be the absolute end of any academic career. Now some academics are arguing for an amnesty on the basis that everyone did it and others are ignoring it.

Edit: I should point out like the other commenter that a lot of the fraudulent data is laughably blatant. Like pulled ‘random’ numbers straight from their head, not even using any type of random number generator. Or pulling random numbers from wrong distribution. Or not using realistic noise etc.

[mike_hearn]

> Its more likely that a random individual's opinion about a random niche of scientific study will be wrong than that of an individual trained on that niche

Well ... this intuition should definitely be true in theory, but in practice some fields are sufficiently distorted by biases (financial, ideological) that it's possible to be more correct as a random individual than an individual trained on a niche. The random Joe Schmuck may or may not have less knowledge, but they definitely have less career-related bias and that is often decisive.

There are plenty of examples where ordinary people manage to systematically beat the experts. A good recent example was the COVID lab leak theory. The entire field of virology and public health came down on one side of that issue, even though it was obvious that a lab accident was highly plausible, and then it turned out they all knew were all lying through their teeth and did it deliberately anyway. The rando nobodies who said "yeah, well, I don't trust the experts" came out ahead and this is now widely acknowledged. In the UK both Boris Johnson and his at-the-time health secretary have admitted that it is the most likely explanation for where COVID came from.

Ultimately, training is of no use if incentives cause people to mislead.