[camelNotation]

Randomization solves for the problem, that's true, but that isn't what people are arguing when they criticize social science research like this.

Theoretically, if you have X number of factors you can't control for, then there should be some threshold Y where the sample size benefits from random selection and accounts for those factors. Right now, social scientists have formulas they use to establish what is and is not an acceptable population when doing experiments like this.

The problem that is being identified (and can't be stated often enough, honestly) is that human emotional preferences and variability of experience means that X is far more variable than anyone can conceive. A human being is an extremely complex biological system and that complexity compounds when you begin comparing people to one another and in groups. So unless you are actively controlling for every possible factor in that system, the assumption that you can use any formula to establish a reasonable population size Y is absurd.

Anyone with half a brain can see this, but social science has to deliver on a product and justify its existence with research funding, so it developed standards - like the aforementioned formulas - that allow it to ignore the problem of identifying X and instead just assuming their synthetic Y will do.

This is the real reason why psychologists have a replication crisis and why they always will. Many of them don't even use the standards they have because honestly, they all realize it's nonsense to begin with.

[ASpring]

Thanks for this comment, it's well thought out and elucidates what the original poster is probably critiquing.

I want to make sure I'm understanding, are you saying that there is no sample size with which you would be comfortable making a conclusion about this? That's what I'm taking away from this comment "the assumption that you can use any formula to establish a reasonable population size Y is absurd."

[camelNotation]

What I mean is that social science research will only ever be able to speak to the samples they are studying. That's usually okay, though, because typically in an experiment with difficult variability, you can gain certainty through replication. With social science, the variability in human beings is so significant that I don't think replication works the same way. I think you would need to replicate studies about a dozen times across different cultures, timeframes, languages, regions, religions, etc just to even begin approaching something like reliable results.

And given the fact that social science has a replication problem (an understatement if I'm right), the entire area of study is suspect.