[LanguageGamer]

"When asked to draw circles representing themselves and friends or family, for example, people tend to self-inflate their own circle but they self-inflate more in individualist cultures."

This sort of methodology sounds sketchy to me - how much can we really learn from this? Does it reproduce? If it does, how do we know there isn't some other cause?

[michaelt]

I wondered exactly the same thing - how have they controlled for ideographic vs alphabetic writing systems? - so I looked at the paper's citations.

It turns out this paper from Talhelm & Dong supports its methodology by citing a paper by Talhelm et.al. and one by Dong et.al.

[debacle]

> It turns out this paper from Talhelm & Dong supports its methodology by citing a paper by Talhelm et.al. and one by Dong et.al.

This should be the top comment. This thread is chock full of pop sociology, to the degree that I really wasn't sure how to respond to much of it.

[adrian_b]

While other methodological aspects may be more doubtful, in the paper of Talhelm of Dong there was no "ideographic vs alphabetic" problem.

That paper compared Chinese people with Chinese people, where both groups had been assigned randomly and forcibly by the communist authorities to become agricultural workers in wheat-cultivating regions or in rice-cultivating regions.

The only confounding factors could be other geographic differences besides their major crops.

The point of the paper was to exploit this unusual historical fact as a social experiment that has eliminated most confounding factors that exist in other comparisons, like the factor mentioned by you.

[mistermann]

> Does it reproduce? If it does, how do we know there isn't some other cause?

Same way as always in the non-physical realm: we don't. Luckily, perfection may not be required, adequacy may be adequate.

[hennell]

It does sound kinda sketchy, but the preceding sentence to your quote mentions it 'has been shown in earlier work' so presumably there are some studies somewhere showing the experiment, links and how much we can learn from it etc.

It still might have flaws, but it's not like they just got people to draw charts and interpreted it as 'collectivist' and 'individualist' for the first time in this study.

[throwup238]

There’s a replication crisis in sociology and psychology and these stupid drawing tests are exactly why.

They’re just reusing flawed techniques from flawed research.

[layer8]

Shouldn’t the replication crisis cause the reuse to fail? ;)

[abdullahkhalids]

A story by Feynman [1]

> ... Other kinds of errors are more characteristic of poor science. When I was at Cornell [1945–1950], I often talked to the people in the psychology department. One of the students told me she wanted to do an experiment that went something like this—I don’t remember it in detail, but it had been found by others that under certain circumstances, X, rats did something, A. She was curious as to whether, if she changed the circumstances to Y, they would still do, A. So her proposal was to do the experiment under circumstances Y and see if they still did A. I explained to her that it was necessary first to repeat in her laboratory the experiment of the other person—to do it under condition X to see if she could also get result A—and then change to Y and see if A changed. Then she would know that the real difference was the thing she thought she had under control. She was very delighted with this new idea, and went to her professor. And his reply was, no, you cannot do that, because the experiment has already been done and you would be wasting time. This was in about 1935 or so, and it seems to have been the general policy then to not try to repeat psychological experiments, but only to change the conditions and see what happens.

[1] https://gwern.net/maze

[ordu]

> This was in about 1935 or so

It is not so anymore, psychologists are taught to design experiments showing that changes in an independent variable correlate with a dependent variable. I'm not sure when the change came, but I'd guess that in 1970s it was so already.

[Terr_]

*taps finger to forehead* You can't have a replication crisis if every paper is a new thing without retesting old conclusions. :p

From my brain's obsessive make-a-fun-analogy circuit:

1. Our research shows we can count umbrellas in aerial photos to predict future rainfall. This Predictive Aggregate Umbral Coverage will revolutionize climatology!

2. Our new research using PAUC [1] shows the country of Elbonia will become a desert in a decade.

[mmcdermott]

If the experiment is re-run, but it'd never be noticed if it the results were simply cited and taken as proved.

[metalspoon]

Their first paper on rice theory was cited like 2000 times, so you can check if anybody debunked this theory. I haven't.

I was also surprised to see that this theory has been published in reputable journals (like Science and Nature Communications). So, odds are that critics haven't made strong arguments on this theory so far.