The amount of armchair criticism of this sort of thing always surprises me.
There's literature on the the number of response options and it basically says:
People respond quicker with binary options.
You lose information though. People get pissed there's no nuance available (no "maybe" for example). They refuse to answer because they can't say anything other than thumbs up or thumbs down.
As you increase the number of options, your ability to predict increases. Predict the same thing later, predict other things, etc. It stabilizes though.
It does take longer for people to respond, but then they complain about what "slightly agree" means.
Regardless, though, you can ask people to do it, and if they just do it, you can predict from it.
Even when you let people omit responses and refuse to respond, you can model that refusal to see what it predicts and then use their refusal anyway.
You can model how people use the response options in different ways, and in general it doesn't matter too much.
You can even eliminate rating scales altogether and use entirely different systems (forced-choice between different statements), but those don't actually work much better either.
Yes, you can always respond in a cheeky, subversive,or manipulative way, but then that's an entirely different issue altogether. You can always do that, and all you've shown is that you're smarter than a rating scale?
As for the original article... these criticisms have been made for decades. Decades. These are my general impressions:
1. The problem of western focus in behavioral sciences is a problem of western focus in the sciences period. Many behavioral scientists would love to do research across multiple sites but cannot afford to do so. And this doesn't stop all sorts of other biomedical research from being done on very narrowly defined groups of people (or animals).
2. Effects observed in undergrads generalize a lot more than the author is letting on, and effects observed in western populations generalize even more. I'm not saying it's not important to study things cross-culturally, only that the idea that people are fundamentally different in different settings is itself flawed. Many things have been examined across different cultural settings, and the differences are not all that dramatically different. In fact, in one recent replicability study, the effect of culture/sociogeographic population was one thing that didn't seem to matter that much. Some studies replicated and others didn't, but the sociogeographic setting didn't seem to matter very much.
I agree that being more sensitive to human variation is critical, but like a lot of things with behavior, there's a lot of grey areas, which people don't like to hear.
There are whole fields within psychology and the behavioral sciences devoted to these issues. People have put a ton of effort into studying them, considering all the issues being raised here as well as many others to numerous to count, and it's like all that work gets brushed aside like snow in the wind because of random blog posts and anecdotal experience.
What people complain about is maybe not the important thing. Its getting actionable results.
And what about the 'Have you quit beating your wife' part? Where I don't lie on any part of the spectrum they've drawn. Ok, that's the same with thumbs-up-or-down, but with the spectrum its in your face that you have no answer that's meaningful. I refuse to answer those kind far more often that thumbs-up.
Finally, mega-corporations wanting actionable results are not 'brushing aside with anecdotal experience'. That's the point I made. They're spending billions and want results, and more often use thumbs-up-or-down. That trial has five orders of magnitude more data than all the graduate students in history added together.
The amount of armchair criticism of this sort of thing always surprises me.
There's literature on the the number of response options and it basically says:
People respond quicker with binary options.
You lose information though. People get pissed there's no nuance available (no "maybe" for example). They refuse to answer because they can't say anything other than thumbs up or thumbs down.
As you increase the number of options, your ability to predict increases. Predict the same thing later, predict other things, etc. It stabilizes though.
It does take longer for people to respond, but then they complain about what "slightly agree" means.
Regardless, though, you can ask people to do it, and if they just do it, you can predict from it.
Even when you let people omit responses and refuse to respond, you can model that refusal to see what it predicts and then use their refusal anyway.
You can model how people use the response options in different ways, and in general it doesn't matter too much.
You can even eliminate rating scales altogether and use entirely different systems (forced-choice between different statements), but those don't actually work much better either.
Yes, you can always respond in a cheeky, subversive,or manipulative way, but then that's an entirely different issue altogether. You can always do that, and all you've shown is that you're smarter than a rating scale?
As for the original article... these criticisms have been made for decades. Decades. These are my general impressions:
1. The problem of western focus in behavioral sciences is a problem of western focus in the sciences period. Many behavioral scientists would love to do research across multiple sites but cannot afford to do so. And this doesn't stop all sorts of other biomedical research from being done on very narrowly defined groups of people (or animals).
2. Effects observed in undergrads generalize a lot more than the author is letting on, and effects observed in western populations generalize even more. I'm not saying it's not important to study things cross-culturally, only that the idea that people are fundamentally different in different settings is itself flawed. Many things have been examined across different cultural settings, and the differences are not all that dramatically different. In fact, in one recent replicability study, the effect of culture/sociogeographic population was one thing that didn't seem to matter that much. Some studies replicated and others didn't, but the sociogeographic setting didn't seem to matter very much.
I agree that being more sensitive to human variation is critical, but like a lot of things with behavior, there's a lot of grey areas, which people don't like to hear.
There are whole fields within psychology and the behavioral sciences devoted to these issues. People have put a ton of effort into studying them, considering all the issues being raised here as well as many others to numerous to count, and it's like all that work gets brushed aside like snow in the wind because of random blog posts and anecdotal experience.