[Aurornis]

> We found that high-dose supplementation was positively associated with 3 of 11 functions assessed: verbal memory, visual memory, and flexibility or set shift. However, the association with flexibility or set shift was not significant after correction for multiple testing.

They tested 11 different measures. Only 3 of those tests showed significance, but after they corrected for multiple testing the significance of 2 of those disappeared. Only 1 test remained below the significance threshold.

This is on a set size of about 500 children who completed the study, randomized to the two groups.

The only measure that remained statistically significant was visual memory. If you look at Figure 2 it's not even clear that there was a trend toward improvement because everything is so scattered, including a couple measures that were trending worse with higher Vitamin D levels.

This study isn't very convincing. It's a classic p-hacking trick to include many different smaller tests so if one of them pops up as an outlier you can claim that something was significant.

[marcosdumay]

> It's a classic p-hacking trick

It's a hypothesis seeking study. It just invalidated 8 of them and picked 1 ok-ish candidate to run an actual study in.

The only thing wrong here is there's only one format for submitting a paper.

[medymed]

And if the 1 ok-ish candidate study turns out to have significant results, publication bias has to be considered: that dozens or more other studies of vitamin D (or whatever) may have not found significant results of previously ok-ish hypothesis candidates and those negative studies are often not published.

I think this is quite the case for vitamin D which has multiple physiologic roles and is studied extensively for relation to many categories of health issues. One more reason why it can be stunningly impressive when something/anything health-related is eventually proven.

[dlcarrier]

Vitamin D is a also an indicator for sun exposure, which has a myriad of health effects, so vitamin D levels are often correlated with a lot of unrelated variables that need to be accounted for.

[Aurornis]

> It's a hypothesis seeking study.

No, it was not.

It was a post-hoc analysis of a different study:

> This is a post hoc secondary analysis of the blinded, placebo-controlled Copenhagen Prospective Studies on Asthma in Childhood 2010 RCT conducted from March 4, 2009, to November 17, 2010.

Also to address this point:

> It just invalidated 8 of them and picked 1 ok-ish candidate to run an actual study in.

I don't quite know what you're talking about, since there were 11 measures looked at in this analysis and many of those measures were somewhat overlapping (various memory tests), so it's dubious to claim that the 1 significance they found means something as opposed to being an outlier they cooked up after shaking up the study data enough different ways.

[jampekka]

Also it was an asthma prevention study, not cognitive functioning one, adding even more "researcher degrees of freedom".

Doing such side studies is fine in itself, but selling such shakey results as "This study suggests that high-dose vitamin D3 supplementation during pregnancy may be associated with improved cognitive functioning at age 10 years." is a stretch.

[letmevoteplease]

Where are you getting "after they corrected for multiple testing the significance of 2 of those disappeared"? The text you quoted says, of the three, one disappeared and two remained.

[margalabargala]

Relevant xkcd

https://xkcd.com/882/

[sushid]

https://www.youtube.com/watch?v=42QuXLucH3Q