|
Okay, this is really interesting. I'm pretty skeptical of their methodology for assessing "statistical training." The setup could result in studying a different correlation entirely. For example, we might instead be studying the difference in a researcher who's unwilling to send a survey back without Googling or double-checking the work vs. a researcher who is busy or willing to be wrong... among researchers who are willing to respond in the first place. I don't expect that to be a productive line of discussion, though, so I'll try to ignore it. I can't help but think of Thomas Kuhn, who argued that the institutions of science (from researchers to reviewers to the press) tend to ignore study results that conflict with the current paradigm. So as long as our paradigm is correct, science progresses extremely quickly. When our paradigm and assumptions are wrong we spend a lot of time floundering because we ignore conflicting data that should instead lead to refinements or more questions. We want to slap a label of right/wrong on something and move on to the next question. That mentality can really hinder our ability to assess anomalies later on. This is similar to the problem discussed in the paper. When researchers decide a paradigm or test result is true or false with no further thought to the confidence level or details, you end up with a system where uncertainty and anomaly are both ignored. It then takes a tremendous amount of momentum to overcome the assumptions, which are sometimes several steps back into "accepted science" at that point. We might even throw out some good ideas from the previous paradigm as we transition. I'm not a physicist, but it seems like they're seeing this happen right now. We've certainly seen this several times with dietary health. A decent analogy might be a hike with unclear trails and markers. At the first crossroads we might decide that left is the correct path to your destination. Once down that path, there are dozens of other paths. If we find trails that continually lead nowhere, the common human response is to keep trying well past the point where we should have gone back to our original assumption. When we finally decide to go back and try the right-side path we completely give up on the left side, despite the fact that we haven't checked every possible sub-trail on the left side. Wandering may be impossible to avoid, even with good judgement, but we can avoid a lot of wasted time by looking back at each of our crossroads/assumptions, assessing the probability that each is correct, then moving forward in a way that's most likely to answer a new question while testing the previous assumptions. By and large this is not how science is working. We get a few studies, accept something is true, and then wander off randomly down the trails that look most interesting. |