[pdonis]

> RCTs are used to establish causality and are as much “proof” as you’re gonna get in science.

No, they're not. The real "gold standard" in science--the standard that prevails in, for example, physics or chemistry--is a controlled experiment. Not just a "randomized controlled trial", but a controlled experiment, where you can actually dictate exactly what state the things you are going to experiment on start out in. And the eventual output of controlled experiments is a predictive model--a model that can predict, accurately, what will happen if you run further experiments. That is what it takes to truly "establish causality".

But in most other domains, including the one under study here, controlled experiments simply cannot be done and predictive models with any kind of accuracy simply don't exist. The correct response to that unfortunate fact is to realize that we can never achieve the same level of confidence in these other domains as we can in domains like physics or chemistry where we can do controlled experiments. Unfortunately, the response "science" has settled on instead is to pretend that it doesn't matter--that because we can't do controlled experiments in these other domains, the universe will somehow magically lower its standards of what it takes to achieve the level of confidence we want. But the universe doesn't care what we can or can't achieve.

[asdff]

They did do a controlled experiment here. They had articles in a treatment group and articles in a controlled group. What are the shortcomings you have in mind when you say that "controlled experiments simply cannot be done" when it appears that they have done a controlled experiment?

[pdonis]

> They did do a controlled experiment here.

No, they didn't. You can't do a controlled experiment on humans. Nobody has a "human source" that can stamp out a series of humans that are identical in all respects, to be used in an experiment, the way physicists have "particle sources" that can stamp out a series of identical particles. That's what "controlled experiment" means. The fact that they call one group a "control group" does not mean it's a controlled experiment. Humans can't be controlled to the degree required.

[asdff]

At that point you might as well write off the entire field of biology since no two animals could be identical. Even clones could be subject to random point mutations.

[pdonis]

> At that point you might as well write off the entire field of biology

No, you don't need to write off the entire field, you just need to be aware of its limitations. As you should be with any field of knowledge.

> no two animals could be identical. Even clones could be subject to random point mutations.

Yes, and any honest assessment of what we know in biology, and how confident we are in our knowledge, has to take these things into account.

[DiggyJohnson]

Because the experiment itself isn’t closed.

[asdff]

What do you mean by closed?

[pdonis]

I think he means that you can't isolate and control the environment of humans the way that physicists or chemists can isolate and control the environment of particles or molecules that they are experimenting on. That's an additional issue to the one I raised in another response just now in this subthread.

[asdff]

You can when you have many samples divided into treatment and control groups that are otherwise identical, that overcomes any biases or confounding variables that might be influencing the design and ensures that what you are seeing is causal.

How much are physicist or chemists really controlling in the lab setting? There could be plenty of confounding variables in their experiments too. Maybe "RT" in this lab for that publication for that experiment is actually 75*F and its 71*F in your lab, or you are at different elevations. Maybe no one calibrated the instruments for years. Maybe the reagent wasn't fresh and absorbed too much moisture or oxygen from the room. Maybe an undergrad dropped the balance on the floor and was afraid to tell anyone.

To overcome those potential confounding variables and other biases, chemists and physicists often turn to the exact same statistical tests being employed by people in the social sciences. Technical replicates are the norm in hard scientific experimental design because of how many biases could be present in the laboratory. It's a chaotic environment. Good experimental design builds robustness no matter what your topic is.

[pdonis]

> How much are physicist or chemists really controlling in the lab setting?

A lot more than can possibly be controlled when you're gathering data from events in the real world instead of in a controlled lab environment.

> To overcome those potential confounding variables and other biases, chemists and physicists often turn to the exact same statistical tests being employed by people in the social sciences.

No, not "often"--"when they have no other choice". The preferred method of dealing with such variables is to measure them, develop predictive models for how they affect the desired outputs, and test those models in further experiments. For ewxample, if "nominal" temperature is 75 F but temperature in labs can vary, physicists or chemists will want to do experiments over a range of temperatures, develop a predictive model for how temperature affects the results, and test the model. They won't just throw up their hands and do "statistical tests" and call it a day--unless it's impossible to do anything else. Which it almost never is in physics and chemistry.

> Good experimental design builds robustness no matter what your topic is.

And good science is aware of the limitations inherent in each specific field no matter what the field is. Good science does not make claims that are not justified in the light of the limitations of the field. "Statistical tests" simply cannot give the same level of confidence as predictive models that have been tested against further experimental data and have passed the tests. And good scientists should not pretend otherwise.

[DiggyJohnson]

Thank you, I didn’t have the time to put together a succinct paragraph like yours.