[thejacenxpress]

Study 1: "Students participated in exchange for the chance to win one of three prizes valued at $700."

Study 2a: "Two hundred and sixty UBC students participated in exchange for entry into a lottery or for course credit (78% female)"

Study 2b: "In Study 2b, 518 adults were recruited from a science museum in Vancouver, Canada"

Study 3a: "In Study 3a, 242 UBC students participated in exchange for course credit or candy (74% female)"

Study 3b: " we were able to recruit a very large sample of UBC students, who participated for course credit (N=2303; 76% female).

Study 4: " we recruited our sample through the GfK Knowledge Networks Survey Panel. Panel members respond to an average of two online surveys per month and receive small cash rewards and prizes for survey completion"

Make of it what you will

[gk1]

Whenever I see "in a study of N people..." I translate that to be "in a study of N university students..."

[mintplant]

I noticed this a lot when reading through studies on visual working memory for a paper. In one, the participants were the two lead authors and the grad students in their lab.

[mercer]

Yes, and that's not always a problem. I remember this issue was addressed in my 'methodology' classes where it was explained that a lot of research has been done to figure out in which ways student/non-student participants differ.

[elliotec]

What do you make of it?

[thejacenxpress]

small unrepresentative sample size. a problem many studies have issues with.

[TeMPOraL]

Actually, compared to most studies I'm used to, this is a commendably high sample size. 2a and 3a have N=~250. 2b is 518, and 3b is over 2000. When was the last time you saw a psychological study with sample sizes this big?

[wldcordeiro]

Aside from just size though the major problem is the gender disparity of ~80% women in most of the studies. I didn't see mention of ethnicity but I imagine it's overwhelmingly white.

[TeMPOraL]

Yeah. This is a big problem, because your average psychology student is a very particular kind of person - young, middle class woman that is bad at maths and simply wants to get a degree from something.

Before I get called out as -ist, let me explain. Psychology is one of the few "default" subjects - i.e. something you pick when you don't know what do you want to major in. It's also a stereotypical feminine subject, hence huge gender gap. The other popular "default" choice is economics, which tend to draw those proficient at maths.

The point being, psychology students are a very particular subset of the population, and thus it's difficult to generalize results from them to everyone.

[lmartel]

I agree with the unrepresentative part, but absolute sample size numbers don't tell you anything. For some experiments, N=100 is more than enough, and for others N=10000 isn't even close. Did you look at the p-values or any other confidence measurements?

[dclowd9901]

Statistically, doesn't a sample size of 2000 get you pretty close to +/- a 5% margin of error?

[queeerkopf]

I'm not sure what you mean by 'statistically' above.

In general you can't say whether a sample size of 2000 will get you within a specific margin of error without additional information. It very much depends e.g. on what the hypothesis you're testing is or what you're trying to estimate.

[mattmanser]

And all are students, who are going to be generally middle class and young.

[jbackus]

This is very common across most psychology studies from what I understand.

> In the Journal of Personality and Social Psychology, the premier journal in social psychology – the subdiscipline of psychology that should (arguably) be the most attentive to questions about the subjects’ backgrounds – 67% of the American samples (and 80% of the samples from other countries) were composed solely of undergraduates in psychology courses (Arnett 2008). In other words, a randomly selected American undergraduate is more than 4,000 times more likely to be a research participant than is a randomly selected person from outside of the West.

Source: http://hci.ucsd.edu/102b/readings/WeirdestPeople.pdf