[amadsen]

The whole thing is just a big straw man argument. Well, actually the piece is pretty incoherent and I found it hard to find a red thread through the musings on such diverse topics as WIMPs, pentaquarks and the functioning of drift chambers. But the core of it seems to be at she thinks that scientists are now sitting on massive piles of data, mine through it looking for any oddities and then jump on the chance to publish a discovery of a new physical phenomenon as soon as one is found.

I can't judge whether she is just ignorant or deliberately misleading (she claims to have worked in physics), but this is just not how experiments are done. You almost always start with a theoretical prediction by a new theory, and there is no lack of theories out there so it has to be a pretty good one that has plenty of justification to merit attention enough to invest the 10+ person-years of effort it typically takes to conduct a rigorous analysis of a chunk of data from a modern large scale scientific instrument. You carefully demonstrate in various side bands your ability to correctly model every background process and instrumental effect that could influence the measurement. After that, if the data very clearly favors the new prediction over the null hypothesis, you may publish the discovery in order to invite other scientists to confirm or refute it. This does not mean that you or your peers accept that the new theory is correct.

And this is all supposedly in support of the thesis that large experimental facilities are a waste of money, which is patently nonsensical. They have been estimated to pay for themselves in direct returns alone (training of students and researchers, spin-off applications to silicon sensors, cryo technology, vacuum technology, lasers, accelerators, computing, etc). And the long term importance of investments in fundamental research are incalculable. For perspective, remember that the electron was considered a "useless" discovery back in 1897.

[blix]

It's a bit of a ramble, but it's hardly incoherent. What seem to like diverse topics to a physicist are actually not all that diverse when looking from a holistic perspective. It's not that scientists are sitting on huge piles of data and mining them for interesting results without testing hypotheses (though we are doing exactly this in many domains), it's that the fundemental approach to statistical analysis is flawed.

With enough data, good enough filters, and a wide selection of adjustments for background processes, any model can be made to work. Putting the blocks together is fundementally an excercise in bias, and truly limiting this bias requires significant discipline that is highly disincentivized and therefore uncommon.

The way she writes invites dismissal from working scientists due to its imprecise and conversational style, but I think she makes many salient points about the flaws in the modern scientific institution.

[amadsen]

> the fundemental approach to statistical analysis is flawed

This is a very strong assertion. What changes do you think are needed?

> With enough data, good enough filters, and a wide selection of adjustments for background processes, any model can be made to work.

Sure. Which is why no scientist will care that a particular signal model "can be made to work". You try everything you can to explain your data with only background processes and only if this fails do you consider alternatives. The more adjustments you allow, the harder it becomes to favor signal over background.

> Putting the blocks together is fundementally an excercise in bias, and truly limiting this bias requires significant discipline that is highly disincentivized and therefore uncommon

Another wild accusation without evidence. Why do you believe this is true? And if it is, where are all the false discoveries? In a small team with less oversight, I'm sure cutting corners happens. In a large experiment like those discussed here? No way. The embarrassment of having to retract a false discovery is a pretty strong incentive to ensure the integrity of results, and they have enough internal controls to enforce it.

[blix]

It's not a "wild accusation" or personal attack, it's a fundemental truth about the nature of modeling. You can do better against it if you start off by recognizing that it is there. Nothing is unbiased.

The embarassment of retraction is also a strong incentive to not challenge the status quo. If everyone is using the same bias and assumptions no one has to worry about retraction. The more small teams you have, the easier it is for different biases to live; the more centralized, the easier it is to succumb to groupthink.