The series of tweets for me just wasn't illuminating and I didn't get what the actual 'paradox' was given the graphs. But my issue was more that the graphs weren't clear in pointing out what I should be looking at.
If anyone's impatient like me, this example from wikipedia helped me get it:
> For example, a person may observe from their experience that fast food restaurants in their area which serve good hamburgers tend to serve bad fries and vice versa; but because they would likely not eat anywhere where both were bad, they fail to allow for the large number of restaurants in this category which would weaken or even flip the correlation.
No, you aren't the only one. It has become even worse now that Twitter will not render without JavaScript enabled. Unfortunately, I still do not know what Berkson's Paradox is because I will not enable JavaScript for Twitter.
The latter appears when analyzing subgroups gives a different result than analyzing the pooled data.
The former is about correlations that appear in samples which are not representative of the general population, due to the way that those samples are selected.
> The latter appears when analyzing subgroups gives a different result than analyzing the pooled data.
> The former is about correlations that appear in samples which are not representative of the general population, due to the way that those samples are selected.
You just said the same thing twice. Think about it.
For one you used terms like "subgroups" and "pooled data" and for the other "samples" and "general population". Those are the same things.
Then you used "[the effect] appears in" and in the other "correlations". Well, Simpsons paradox can also manifest itself in correlations. So you just said the same thing twice.
Simpson's paradox: analyzing trends per subgroup can give a different result than pooled data.
Berkson's paradox: analyzing a single subgroup selected with a function aggregating two traits (additively?) will indicate an anticorrelation between the traits.
Simpson's paradox says you can't judge group trends from subgroup trends. Berkson's paradox says given a group selected in a specific way, it will have a certain property in itself. They're just different statements.
I think Berkson's paradox is more specific than just correlations arising from non-random sample selection. Correlations that are not representative of the general population could still be useful, if it's a meaningful correlation within some subgroup of interest. The problem is when the features you are correlating relate too closely to the features that were used for sample selection - then you can end up with a trivial result.
I've always learned of Simpson's paradox as relating more to different sample sizes when partitioning data, which can happen entirely arbitrarily - for example a baseball player getting injured part way through the season.
The fact that one player's at bats get partitioned differently than another's is not caused by the on field performance, so there's no "double dipping" going on like I would imagine with Berkson's. Conversely I'm having trouble fitting a Berkson's example into the framework of Simpson's paradox, since there's no reason the poorly-selected subpopulation can't theoretically be exactly half of the general population. And if all of the samples are of equal size Simpson's paradox doesn't exist anymore (because with equal bin sizes the mean of means is equivalent to the overall mean).
In the first one you have a partition in subgroups A and B (or more than two) which show similar correlations, different from the correlation seen in A+B.
In the second one you have only a subgroup A (the implicit complement notA is not observed) where the correlation is not the same as in the (unobserved) full population A+notA. Nothing is said about the correlation in notA. It could be at either side of the correlation in the full population, while in Simpson’s paradox both subgroups are in the same side.
Edit: and I also mention "due to the way that those samples are selected" for Berkson's paradox where the selection is based on the variables of interest while in Simpson's paradox the subgroups are "external" (but influence the correlation between those variables).
You're not alone. I think it caught on because a long article (even with pictures) might seem like too much of an investment to a lot of people but a self-contained tweet that keeps getting extended is less intimidating.
TBH, I'd say it's less that I dislike this form of presentation than that I hate all the anti-pattern bloat that Twitter adds, like clickable items not being detectable by extensions and previews being cut off.
Yes it's one of the reasons I hate Twitter. It was designed with aversion to substance. Personally, I find older fashioned forums (with small communities of experts) more illuminating.
Wikipedia was much clearer for me, https://en.wikipedia.org/wiki/Berkson's_paradox , but ymmv of course.
Another good statistical foible to be aware of along with Simpson's.