[zosima]

Though p-values are probabilities.

They are the probability that the data seen (or more extreme) in the experiment were generated given the null-hypothesis is true.

Now, of course to fully understand the p, you also have to understand the null hypothesis. And yeah, sometimes it is misspecified. (by e.g testing out many null-hypotheses and only showing the more interesting ones, or accidentally creating a bad unlikely null-hypothesis which may allow for many uninteresting alternative hypotheses.)