[danepowell]

Most comments here point to cherry picking and "p hacking" as being the primary problems with p values. Certainly those are major issues, but I think they miss the real point of the article, which is that null hypothesis testing is fundamentally broken, or at the very least doesn't do what most people think.

A simple example of this can be shown with the following pair of tests:

Testing for a fair coin:

  - Null hypothesis is you have a fair coin
  - You observe 100 heads in a row
  - Given a fair coin, it's extremely unlikely to observe 100 heads
  - Therefore it's not a fair coin

Okay, that makes sense, but this is logically the same as:

Testing whether a person "Bill" is an American:

  - Null hypothesis is Bill is an American
  - You observe *Bill is a US congressman*
  - Given Bill is an American, it's extremely unlikely to be a congressman
  - Therefore he's not an American

Obviously that's some broken logic, but it's a perfectly valid way to get p < .05

[mokus]

The problem in the second case is not null-hypothesis testing, it’s sample size.

[mar77i]

What about the fact that the probability of a non-American getting elected to congress being rather low?