[gwern]

This is definitely a study and science. It's actually a rather high quality experiment too, I was in awe of the way they carefully constructed the pairs of decisions to make them as identical as possible, and then got the same law student to write an article for each member of a pair before randomizing to ensure they were written as similarly as possible. Everyone knows that you should be blocking to maximize statistical power, but very few people ever go that far. (And that's probably a good part of why despite the relatively small _n_, they still get very clear causal effects.)

[stevage]

I still don't understand why they bothered writing the articles for the ones that weren't being uploaded. I'm sure there's a reason, it just wasn't obvious to me.

[gwern]

Because that threatens the validity of randomization and the comparability of the counterfactual, if the article writer knows it'll be uploaded, they can do things like pick and choose which ones to work hard on. "Oh, I like this case, it accords with my personal politics, so I will work extra hard on it since I know it'll be uploaded." (One example from the RCT literature of how knowledge of the randomization before the intervention can be a bad thing: in one early study of steroids for babies, the hospital nurses 'knew' that steroids helped, and so they would pull out the randomizing ball and if it was 'wrong' for that baby, they'd put it back in and try again. This is why clinical trials try to use pregenerated randomness if they can't blind the nurses.) Even if mechanisms like that don't add a systematic bias, they do add noise and reduce the statistical power to detect an effect.

It also helps the ethics angle if you are simply holding back articles which you will upload eventually.