[scishop]

No.

In Frequentist thinking; p=0.05 means that if there was in reality no difference in your A and B and you repeated the experiment many times, 5% of the observed differences would be equal to or greater than the difference you just measured.

No probabilistic statement about the results being correct or incorrect can be made from a Null-Hypothesis significance test.

[allforJesse]

(apologies for repeating from above)

If you have a few minutes to spare, I would very much welcome your thoughts so that I can either correct the article, or take it down - The last thing I want is for it to sit out there on the open internet as misinformation.

My goal was to create a framework which — while less mathematically accurate (hence “rhetorical device”) — helped convey the seriousness of making business decisions based on P = 0.05 to people for whom 95% statistical significance doesn’t mean anything. And clearly, based on reactions here, I failed at that goal.

So, if you’re game, I’ll quickly to walk you through my thinking, and you can help me understand where I went wrong. Best way to contact?

[nickm12]

+1 Well said.

This is correct and the original post is wrong.

[pcrh]

What is the difference between what you wrote and saying "95% of the observed differences would be equal to or less than..."

The p-value (in my understanding) makes a prediction about what would occur if the experiment was repeated infinity times.