[cup]

The downside of this though is that unexpected positive results sometimes get buried. For instance, clinical trials have to report all adverse effects but not positive effects. We've had drugs tested for symptom X fail to have any effect, while causing bald people to have their hair grow back. Yet the company we were testing it for wasn't interested and those results never made it into the public domain.

[rickycook]

i would assume that the positive results still get reported somewhere in the paper? so other people could still pick up on it in the future.

if they’re positive enough then the drug company will fund another study with that as the primary outcome. that seems prudent too: it should be the primary thing you’re examining so that you can design the study correctly, rather than a simple “oh and by the way” side note

[jonathankoren]

That wouldn't happen. You would simply declare, that you want to check if the drug does cause hair growth, and then run a test explicitly looking for that.

Unless you're looking for the effect from the outset, you can't be sure that what you saw wasn't actually random. There's a term for what you're describing, it's p-hacking, and it's explicitly the very thing declaring what you're looking for before you run the test is designed to prevent.

Like many science related topics, there's a XKCD about p-hacking that describes a similar scenario to your example: https://xkcd.com/882/