Ah I see, apologies, I understand your concern.
If we define 'big' and 'small' as being above or below the average (mean) value, then you're quite right that it breaks down for many cases, like the one you gave. But if we define 'big' and 'small' relative to the middle (median) value of the distribution, then it becomes a robust result. Hope that helps!