[mlyle]

> If you average any trial out in a large population there will be "noise", but these people who live with the "noise" are the ones affected and suffering.

If you do a trial in a large population of a drug, device, or clinical practice that does nothing- a perfect placebo- you'll see a variety of effects: statistical noise.

If you do a trial in a large population of a drug, device, or clinical practice that has an effect-- you'll measure that effect, plus the statistical noise.

You can't generally tell for any individual whether the drug helped or hurt. But you can tell that more people did well (or badly) in group A than group B.

You can't even really know exactly how big the effect is precisely: just a range of likely effect sizes.

The more things you try to measure to more precisely zero in on an effect, the greater the chance that statistical noise spuriously makes one of these look important (and the larger the effect must be to be reliably measured). https://xkcd.com/882/

[FollowingTheDao]

> You can't generally tell for any individual whether the drug helped or hurt. But you can tell that more people did well (or badly) in group A than group B.

That is what they found in this study, but the OP said it was likely "noise" and had no scientific basis for saying that. My point; saying something is "noise" is a way to look cool on HN and dismiss any finding that does not fit your world view.

[mlyle]

> That is what they found in this study, but the OP said it was likely "noise" and had no scientific basis for saying that.

It's a small finding in both effect size and statistical significance, and prior probabilities count.

Barely statistically significant findings don't change my beliefs much, because the base rate and prior knowledge matter.

E.g. if you show me a p<0.05 finding that ESP exists, I'm going to dismiss it as statistical noise-- even if the study methodology is perfect it's only 10-20x more likely that ESP works than before, and 20x my prior belief of very near 0 is still very near 0.

If you show me a p<0.05 finding that green jelly beans cause acne, after studying all colors-- I don't care at all.

Here, the commenter you replied to-- api-- suggested that the study clearly indicates that there's reason to be concerned about saccharine and sucralose. It raises a general level of concern about other NNS's, but the data is ambiguous and weak. This is a reasonable reading of the study.