[harry8]

Here's a thing I don't understand - no control group.

Surely we've been trying to treat people with severe, long term depression by other means and not had success. This surely wasn't the first treatment that these people tried in trying to treat their long-term misery. How are both of those things taken together not as good as a control group that allows you to draw meaningful conclusions. It really sounds to me (although it would be better to see the quantification in the source docs) that this is a strong enough result to rule out noise.

If your results are strong enough, you don't need a control. If all 12 participants died while undertaking the trial would we say that it needs to be re-run with a control to be meaningful?

The results look strong. Go for pre-registered replication right away. Sod the control group. Everyone with severe long term depression outside the study is already suffering enough to be the control.

[exelius]

That's why it's a canary study. Sure, it looks good, which means it's worth spending the money on a larger study. And yeah, there are lots of people with severe long-term depression, so find some of them, measure their mood over the same time period, and make absolutely sure that the effects you're ascribing to the substance you're testing aren't caused by something else.

You always need a control group because you can't control for every externality over the exact time period the study was conducted. Maybe the weather was especially nice for those 3 months and the control group feels happier as well? We need a way to quantify the results and be able to answer those questions.

Always be careful with the conclusions you draw from studies without control groups. Small studies without control groups are very cheap to run, so you can run the same study 15 times and only report on it when you get the results you're looking for. Not saying that's what happened here, but we have to apply the same standards of proof to all drug studies or else they don't mean anything. The purpose of this study was likely not to prove "hallucinogens can cure depression" but rather "mushrooms are potentially effective and don't appear to have any glaring safety issues".