[jwfxpr]

You cannot dismiss rigorous statistical analysis by arguing it can never encompass the full dimensions of the data. Of course it can't. The map is not the territory; it is a useful way to find our way around it. Ignoring the map is perilous, if not arrogant, even though it is merely a flawed representation of the real truth.

You might argue that a specific study or meta-analysis contains a bias or misinterpretation, but only if you've actually examined their methodology, data, and reasoning. You cannot argue that all studies of complex topics are invalid simply because their topics are complex.

[tomp]

> You cannot dismiss rigorous statistical analysis by arguing it can never encompass the full dimensions of the data.

This simply means it's not rigorous. See Omitted-variable bias - from [1]: The bias results in the model attributing the effect of the missing variables to the estimated effects of the included variables. For example, including gender but not education or hours worked will result in attributing pay differences to gender, but including all relevant variables shows that's gender is irrelevant.

https://en.wikipedia.org/wiki/Omitted-variable_bias

[dwaltrip]

You entirely missed the point. We can never include every single relevant variable to perfectly explain the observation. It's impossible.

This doesn't mean statistics is useless.

This is the meaning of the phrase "the map is not the territory". All models are flawed, but some are useful.

[mseebach]

No, statistics aren't useless, but its usefulness cuts both ways: if you can add one or two relevant variables and almost entirely remove the observation, then statistics tells you that the observation was only there due to omitted-variable bias.

[dwaltrip]

Sure, that's fine. That's part of using statistics to the best of our ability.

[fwdpropaganda]

I think there's some middle ground between saying that the analysis is rigorous and saying it's useless, no?

[dwaltrip]

Indeed. But that's where all the hard work is -- trying to determine how rigorous something is, knowing it certainly isn't perfectly rigorous.

[PurpleBoxDragon]

>You cannot argue that all studies of complex topics are invalid simply because their topics are complex.

If you take a random sample of studies you can make a statistical analysis. You don't need to examine every cow to make an argument that there are no pink cows, but you do need to do a random sample. And that's if you only want to meet the highest standards of evidence. Much lower standards can be far easier to meet.

[fwdpropaganda]

> If you take a random sample of studies you can make a statistical analysis. You don't need to examine every cow to make an argument that there are no pink cows, but you do need to do a random sample. And that's if you only want to meet the highest standards of evidence. Much lower standards can be far easier to meet.

Your comparison of this problem with pink cows shows that you haven't given it two seconds thought. Estimating the number of pink cows in the world is a very simple problem. Determining pay gap is a very very complex problem that starts with defining what the question really is and associated fights between different interest groups which might prefer one or another definition, then goes on to the (social, privacy, and rights) problem of obtaining the data, and moving on into with the data analysis itself which is just hellish if you want to have any semblance of rigour, and finally policy take aways from the analysis which hinges crucially on how you defined the question initially.

Stop adding to the noise please.

[PurpleBoxDragon]

>Estimating the number of pink cows in the world is a very simple problem.

If by 'estimating' you mean a scientific study that tries to answer the question, then it isn't simple at all. First we need a rigorous definition of pink cows. If I dye my cow pink, does that count? What if other people don't agree with my definition? A pig whose skin is pink is considered pink, so should I only rely on hair color? And what counts as pink? Are we only going with stereotypical hot pink? There is a red cow, but it is a really brownish red. Would a brownish pink be enough to qualify as a pink cow?

So once we solved all those problems, we need to come up with a methodology, and it likely won't be the same everywhere. We could make the problem a lot simpler by reducing our search space to say, only cows on ranches in the state of Montana. But to do a global sampling isn't easy.

>associated fights between different interest groups which might prefer one or another definition

To my knowledge (and with no peer reviewed research to back up my view), there is no groups who have a political stake in what counts as a pink cow. So for that reason it is simpler because there aren't political complications.

But you seem to be confusing something. You appear to be talking about studying wage gap. I was talking about studying studies of wage gaps.

So for my plan, it would work like this:

Taking all the studies of wage gap in the last n years, pick x at random. For each of these, determine if each one does or does not account for some factor that impacts pay regardless of gender (say height of employee). You can then compute what percentage of studies took this factor into account.

Then you repeat this with a few other factors, each time repicking the studies investigated. From those percentages, you can determine how often your selection of factors are taken into account, and from that you might be able to make the argument that the data is biased enough to not be usable.

[dominotw]

Isn't that why we have 'replication crisis' in fields that 'can never encompass the full dimensions of the data'

Doesn't replication crisis prove 'that all studies of complex topics are invalid simply because their topics are complex.'

[boomboomsubban]

Science as a whole has developed knowing that it is impossible to encompass the full dimensions of the data, the goal is to find the best explanation given the available evidence.

The replication crisis is a result of the misuse or misunderstanding of the statistics, and the current nature of journals.