[DontchaKnowit]

Can we stop calling stuff like this a "study"? This is not science, this is essentially journalism.

I mean technically yes they are "studying" a subject but this is not a scientific experiment, but it will almost certainly be cited as such just because it's called a study.

[gwern]

This is definitely a study and science. It's actually a rather high quality experiment too, I was in awe of the way they carefully constructed the pairs of decisions to make them as identical as possible, and then got the same law student to write an article for each member of a pair before randomizing to ensure they were written as similarly as possible. Everyone knows that you should be blocking to maximize statistical power, but very few people ever go that far. (And that's probably a good part of why despite the relatively small _n_, they still get very clear causal effects.)

[stevage]

I still don't understand why they bothered writing the articles for the ones that weren't being uploaded. I'm sure there's a reason, it just wasn't obvious to me.

[gwern]

Because that threatens the validity of randomization and the comparability of the counterfactual, if the article writer knows it'll be uploaded, they can do things like pick and choose which ones to work hard on. "Oh, I like this case, it accords with my personal politics, so I will work extra hard on it since I know it'll be uploaded." (One example from the RCT literature of how knowledge of the randomization before the intervention can be a bad thing: in one early study of steroids for babies, the hospital nurses 'knew' that steroids helped, and so they would pull out the randomizing ball and if it was 'wrong' for that baby, they'd put it back in and try again. This is why clinical trials try to use pregenerated randomness if they can't blind the nurses.) Even if mechanisms like that don't add a systematic bias, they do add noise and reduce the statistical power to detect an effect.

It also helps the ethics angle if you are simply holding back articles which you will upload eventually.

[6177c40f]

What about it makes it not science?

[tediousdemise]

Concluding that knowledge influences behavior isn't science. It's common sense.

[bdowling]

A conclusion supported by carefully collected data is science. A conclusion based on common sense is not.

[edgyquant]

You have no idea what science is. Suggesting a hypothesis and then proving it via rigorous testing is what science means. It isn’t whatever the hell you want it to be nor is “science” anything but a method to disprove hypotheses.

[stevage]

They quantified that though.

[edgyquant]

This is a statistical study, what are you talking about?