[PaulHoule]

On top of it there is the awful truth that the ‘scientific method’ is not really practiced.

In junior high school I remember getting taught about the scientific method, particularly the use of controls. I wrote 5 papers and didn’t use a control in any of them (it wouldn’t have been appropriate.).

Even in cases where people obviously should use controls, such as clinical trials, they frequently don’t. There was that paper where they measured vitamin C levels of COVID-19 patients but didn’t compare it to a baseline of people who were not sick, which is problematic in many ways.

When they do meta-analysis by the Cochrane methodology they usually throw out at least half of the studies at the beginning because of glaring methodological flaws. Practically it is not much better than anarchy in terms of what gets funded and published.

[cryptonector]

- Incorrect use of statistics tools.

- Cherry-picking of data.

- Flawed or missing controls.

- Lack of replication, and in the few cases where it's attempted, failure to replicate.

- Non-publication of failures.

- Publish-or-perish providing huge incentives to publish junk.

- Peer review being an old boys club that enforces the party line.

- All funding coming from few sources that tacitly use their funding power to fund only those that toe the party line.

- So much basic science having been done by now that the remaining science to do is generally expensive to do, thus inviting the above funding / control problem.

- Dogmatism.

- Media attention.

These are the problems that plague science today. Some of these have been there for a long time, like dogmatism. There are people alive today who were taught that the continents don't move, and that noticing that South America and Africa fit together and concluding that they must have moved is nonsense of the highest order. There are people alive today whose treatment for Polio was not physical therapy but immobilization. The list of dogma, old and new, is long. The malign ways in which some lords of science fiefdoms defend their dogmas have not gone away in spite of Popper's method.

[gumby]

> Even in cases where people obviously should use controls, such as clinical trials, they frequently don’t.

This is not true in my experience (I have designed and run pharmaceutical clinical trials in humans). Can you give some examples?

Some treatment trials do compare the existing standard of care vs the new one rather than placebo against proposed treatment, but those are certainly controlled studies too.

I do know of cases where a control is impossible, such as some surgical procedures, though even then sometimes sham surgery is performed (this is controversial). Those are singly-blinded controls as the surgeon knows.

[redavni]

It is endemic in statistical studies based on medical data. They reach a conclusion that is statistically supported but never do the work to verify it.

[gumby]

A statistical study is not a clinical trial, which was the term used by the GP poster.

[PaulHoule]

Definitely you want to use controls if you want regulators to take your work seriously. A lot of stuff gets found on pubmed that isn't meant for regulators and it doesn't.

[ethanbond]

Doesn't the question of controls basically boil down to: it's hard to know things without them, but they're often hard to create anyway, ergo it's hard to know things in general - now welcome to the nastiness of real science?

[culi]

Right or the replication crisis. Which people often forget affected far more than just the field of Psychology. It's a problem throughout most of western science in general. And a big problem imo. Because studies are expensive and replicating those studies is nearly just as expensive. What happens when our idealized scientific process is literally just unachievable because we don't have the resources to carry it out

[analog31]

A puzzle seems to be why some scientific fields fare better than others, in the face of the replication crisis. Why do we take Maxwell's Equations seriously? It means that replication, while important, is not the be-all and end-all of science research. There must be something else.

[OkayPhysicist]

Physics (especially classical physics) has the advantage of dealing with things that are both extremely repeatable and consistent, and are just about omnipresent. When a physics model is flawed, you start piling up edge cases that violate our current understanding of physics, until someone comes up with a model that explains most of them while not violating the relatively decent predictive power of the previous models (which obviously have predictive power, since they've been being used to predict things successfully for a while).

For example, Newton laid out classical mechanics. Which very adequately modeled most things moving around in scales we could observe. Except Mercury's orbit was a little fucky. Now, one new model could be that "Classical mechanics works everywhere but on Mercury, where they have different physics", but eventually relativity was postulated, and calculated to predict Mercury's orbit to within our ability to observe it. Rinse, repeat, for things like QM, orbital mechanics, etc, etc.

Heck, there's a pile of inconsistencies in our current models. Neutrinos have mass, the universe's expansion is accelerating, and where's all the antimatter?

[analog31]

Very good points. I think another is that we can develop theories that relate different experimental cases, so that a "replication" doesn't have to be precisely the same study, but different studies in a sense replicate different but overlapping views of the theory.

[lisper]

> the ‘scientific method’ is not really practiced

There is a big difference between "often not practiced" and "not really practiced [ever]". The former is true, the latter is not.