[pdonis]

"because randomized experiments are the gold standard for this type of research, we know the effect we are seeing is causation, not just correlation"

No, they don't. Correlation is not causation, even if you see it in a randomized experiment. With shoddy reasoning like this, it's no wonder science has a replication crisis.

[asdff]

What they did was fine with this setup. You might enjoy reading this article that covers the very perceived issue you are having with the research:

https://towardsdatascience.com/establishing-causality-part-1...

[bjornsing]

As I understand it a randomized controlled trial (RCT) can determine causation. But I don’t think “what they did with this setup” was an RCT. A real RCT in this context should IMHO assign new cases randomly to a “treatment” or “control” group. In this setup they assign legal precedents randomly, but all new cases are effectively under “treatment”. That’s not the same thing.

[asdff]

How are new cases effectively under treatment? They identified 150 decisions, wrote articles for each, and assigned half randomly to treatment or control. they did not consider newer cases beyond this set of 150. An RCT is a type of randomized experiment, but all determine causation:

https://en.wikipedia.org/wiki/Randomized_experiment

[bjornsing]

You’re right. It’s the legal precedents / decisions that are the “patients” here. The new cases (that cite those precedents / decisions) are just… let’s say “test results”.

So yes, this is a way to determine causation in relation to what legal precedents / decisions will be cited in new cases depending on the contents of Wikipedia.

I was under the (perhaps mistaken) impression that they claimed a causal link between their Wikipedia articles and the outcome of new cases. The chosen setup cannot IMHO prove such a causal relationship, in the strict statistical sense. That would require randomizing new cases and applying the “treatment” (new Wikipedia articles) only to some of them.

[Spivak]

https://nulib.github.io/moderndive_book/7-causality.html

https://bolt.mph.ufl.edu/6050-6052/unit-2/causation-and-expe...

https://towardsdatascience.com/establishing-causality-part-1...

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6235704/

https://escholarship.org/uc/item/42v4w8k1

http://ippsr.msu.edu/public-policy/michigan-wonk-blog/random...

https://www.cs.cornell.edu/courses/cs1380/2018sp/textbook/ch...

I’ll leave open the possibility that it’s everyone else that’s wrong but RCTs are used to establish causality and are as much “proof” as you’re gonna get in science.

Hell ya know what I’ll just let the actual paper explain it.

> The second, more important advantage of randomized field experiments is that they can distinguish causation from correlation. The ability to prove causal relationships derives from the combination of two characteristics. The first is having a control group, that is, a group unaffected by the intervention (in our case, publication of a Wikipedia article on the topic) that can be used as a counterfactual to estimate the size of causal effects. The second is randomization, that is, random assignment into the control and intervention groups. With sufficient data and a sound experimental design, the experiment can reduce the probability of being misled by correlation or noise to whatever arbitrarily small value is desired.

[pdonis]

> RCTs are used to establish causality and are as much “proof” as you’re gonna get in science.

No, they're not. The real "gold standard" in science--the standard that prevails in, for example, physics or chemistry--is a controlled experiment. Not just a "randomized controlled trial", but a controlled experiment, where you can actually dictate exactly what state the things you are going to experiment on start out in. And the eventual output of controlled experiments is a predictive model--a model that can predict, accurately, what will happen if you run further experiments. That is what it takes to truly "establish causality".

But in most other domains, including the one under study here, controlled experiments simply cannot be done and predictive models with any kind of accuracy simply don't exist. The correct response to that unfortunate fact is to realize that we can never achieve the same level of confidence in these other domains as we can in domains like physics or chemistry where we can do controlled experiments. Unfortunately, the response "science" has settled on instead is to pretend that it doesn't matter--that because we can't do controlled experiments in these other domains, the universe will somehow magically lower its standards of what it takes to achieve the level of confidence we want. But the universe doesn't care what we can or can't achieve.

[asdff]

They did do a controlled experiment here. They had articles in a treatment group and articles in a controlled group. What are the shortcomings you have in mind when you say that "controlled experiments simply cannot be done" when it appears that they have done a controlled experiment?

[pdonis]

> They did do a controlled experiment here.

No, they didn't. You can't do a controlled experiment on humans. Nobody has a "human source" that can stamp out a series of humans that are identical in all respects, to be used in an experiment, the way physicists have "particle sources" that can stamp out a series of identical particles. That's what "controlled experiment" means. The fact that they call one group a "control group" does not mean it's a controlled experiment. Humans can't be controlled to the degree required.

[asdff]

At that point you might as well write off the entire field of biology since no two animals could be identical. Even clones could be subject to random point mutations.

[pdonis]

> At that point you might as well write off the entire field of biology

No, you don't need to write off the entire field, you just need to be aware of its limitations. As you should be with any field of knowledge.

> no two animals could be identical. Even clones could be subject to random point mutations.

Yes, and any honest assessment of what we know in biology, and how confident we are in our knowledge, has to take these things into account.

[DiggyJohnson]

Because the experiment itself isn’t closed.

[asdff]

What do you mean by closed?

[pdonis]

I think he means that you can't isolate and control the environment of humans the way that physicists or chemists can isolate and control the environment of particles or molecules that they are experimenting on. That's an additional issue to the one I raised in another response just now in this subthread.