[EpiMath]

This is why we report statistical power. It does in fact put probabalistic limits on the size of the elephant. In effect, we can make a statement like "at least 90% of studies like ours will detect an elephant larger than 20 microns if it is actually there on the table". It's not reasonable to interpret the results of a single study without consideration of power.

[blotter_paper]

> It does in fact put probabalistic limits on the size of the elephant.

Emphasis added (obviously). This word was missing from the analysis of whatshisface (GGP). While all of our models are probabilistic, and it would be silly to constantly describe everything this way in normal parlance, a discussion of how we want to go about building and interpreting models of the world is exactly the domain where this needs to be explicitly stated. You're not wrong here, but neither is GP. GGP is wrong, though not provably so (again, obviously).

> In effect, we can make a statement like "at least 90% of studies like ours will detect an elephant larger than 20 microns if it is actually there on the table".

The "in effect" disclaimer makes this statement arguable, but fundamentally we still can't say that. We can only say whether or not an elephant was detected, since detection (in this sense) is subjective. We can't say whether elephants actually exist or not. Perhaps all tables have invisible elephants on them, or perhaps all elephants are hallucinations shared by multiple humans through crazy coincidence. If 50% of elephants that actually exist on tables are invisible, and 10% of visible elephants are not detected due to error, then only 45% of studies will detect an elephant even though it will still seem like 90% to best of our knowledge. Since we don't know what percentage of elephants are currently detectable we need to be even more ridiculous and say something like "we guesstimate that there is a 99ish% chance that at least 90% of studies like ours will detect an elephant larger than 20 microns if it is actually there on the table".

Edit: s/whathisface/whatshisface

[EpiMath]

Yes, okay. There are implicit assumptions in scientific studies ( e.g.: there are no invisible elephants, or the study we are doing is actually related to the question we are investigating! ). Power calculations refer to the model and not to whether it is the correct model. To some extent we routinely worry about certain types of "hidden from observation" problems: we have zero-inflated poisson models, or we worry about what happens if there is a limited susceptible subpopulation ( that could deplete over time differentially depending on treatment, etc ). But it is not correct to suggest that if a study of whatever power does not find an effect, then a huge effect and no effect are equally plausible.

I have seen outrageous examples of "accepting the null hypothesis" many times, but many negative result studies have great value and even a single negative study can provide evidence against a very large effect.

[lutorm]

* There are implicit assumptions in scientific studies ... Power calculations refer to the model and not to whether it is the correct model.*

True, but this crucial assumption needs to be kept in mind when using the study to inform yourself of the state of the world. Too often, it's forgotten that there are two questions that need answering for the study to be relevant and, only one of those has a number associated with it.

In that sense, Bayesian statistics, where this is explicit, are less misleading because they actually draw attention to the fact that we don't know that the model is correct.

[EpiMath]

Agreed.

Interesting point about Bayesian methods, whenever there are potential flaws or additional uncertainty, it's better if they are more explicit to prompt thoughtful interpretation.

[gbrown]

It's still really convoluted and removed from the actual elephant. From a frequentist perspective, the elephant either is or is not on the table, and if it is then it has a certain size - not random quantities subject to probabilistic statements.

If you're willing to treat probability as a measure of uncertainty, then you'd be right at home as a Bayesian.