|
Yeah, I found this article to be annoying AF, because it seemed to fall into the same traps that he's accusing these study authors of making in the first place. It seemed by the end of it he was just trying to yell "correlation is not causation!" but in an even smarter "I am very smart" sort of way. E.g. I certainly found myself agreeing with his points about observational studies, and there are plenty of real-world examples you can point to where experts have been lead astray by these kinds of studies (e.g. alcohol consumption recommendations, egg/cholesterol recommendations, etc.) But when he talked about his reservations re "the wheat" studies, they seemed really weak to me and semi-bizarre: 1. Regarding "The paper doesn't make it easy to replicate its analysis." I mean, no shit Sherlock? The whole point is that it would be prohibitively expensive or unethical to carry out these real experiments, so we rely on these "natural" experiments to reach better conclusions. 2. "There was other weird stuff going on (e.g., changes in census data collection methods), during the strange historical event, so it's a little hard to generalize." First, this seems kind of hand-wavy (not all natural experiments have this issue), but second and more importantly, of course it's hard to "generalize" these kinds of experiments because their value in the first place is that they're trying to tease out one specific variable at a specific point in time. 3. The third bullet point just seemed like it could be summarized as "news flash, academics like to endlessly argue about shit." I think the fundamental problem when looking for "does X cause Y", is that in the real world these are complex systems: lots of other things cause Y too (or can reduce its chances), so you're only ever able to make some statistical statement, e.g. X makes Y Z% more likely, on average. But even then, suppose there is some thing that could make Y Z% more likely among some specific sub-population, but make it some percent less likely in another sub-population (not an exact analogy but my understanding is that most people don't really need to worry about cholesterol in eggs, but a sub-population of people is very reactive to dietary cholesterol). Basically, it feels like the author is looking for some definitive, unambiguous "does X cause Y", but that's not really how complex systems work. |