[penagwin]

That's not what OP is saying though. Let's say p=.001 - That means we're confident the results are not due to chance.

If we repeat the experiment and get a different result, then we need to be looking into confounding variables and testing methodology. Just because the P-value is low doesn't mean there's no fundamental flaw in the experiment used to find the p-value.

[platz]

P values don't talk about the result, they only talk about the probably of the null hypothesis being correct.

You're only allowed to say either nothing happened or it's unlikely nothing happened, but it doesn't say what the "unlikely nothing happened" means.

[kgwgk]

> P values don't talk about the result, they only talk about the probably [probability?] of the null hypothesis being correct.

I would say it’s the other way around.

The p-value says something about the result (how likely this result would be if the null hypothesis were true).

It doesn’t say anything about the probability of the null hypothesis being true.

[Alex3917]

> how likely this result would be if the null hypothesis were true

It's how likely the result is to occur by chance if the null hypothesis is true. A positive result can occur for lots of other reasons even if the null hypothesis is true, and the p-value doesn't tell you anything about how likely you are to get a certain result if the null hypothesis is true (or false).

[kgwgk]

> It's how likely the result is to occur by chance if the null hypothesis is true. A positive result can occur for lots of other reasons even if the null hypothesis is true, and the p-value doesn't tell you anything about how likely you are to get a certain result if the null hypothesis is true (or false).

I don’t think your comment makes sense.

Given a parametric model and a particular value of the parameter (i.e. the null hypothesis) one can calculate the sampling distribution of the data.

Therefore under the null hypothesis the model gives a well-defined probability distribution for the data and you can tell how likely you are to get a certain result.

There is no room for “other reasons”.

[Alex3917]

Let's imagine the simplest case where the scientist never actually ran the experiment in the first place and just made up their data.

Looking at the p-value of a study doesn't tell you anything about how likely the study was to have been based on fabricated data.

[kgwgk]

When we say “how likely you are to get a certain result if the null hypothesis is true”, one should understand “the null hypothesis is true” as “the data is generated by a process perfectly described by the model, including a particular value for the parameter”.

I agree that if the data is made up the results of the study and the statistical analysis based on the results will have no relation whatsoever with the fact that the null hypothesis was or wasn’t true.

The p-value tells you just how likely you are to get a certain (or more extreme) result if the data generating model is indeed correct and the null hypothesis is true.

We agree that the p-value doesn’t tell you anything about how likely it is that the study was based on fabricated data, or how likely it is that the model is correct or how likely it is that the null hypothesis is true.

The p-value doesn’t tell us anything about the real world. It’s a probability conditional on a hypothetical model.