[Aurornis]

> The fact this study even exists is a sign of something having gone very wrong IMHO.

I agree, but for different reasons: The paper is an example of someone sending out surveys to collect self-reports and then writing a paper title as if they had performed a study. They did not. They just surveyed some college students and drew conclusions by running statistical analyses on the data until they got something that seemed significant.

It appears to have worked, though, as I’ve seen it shared across the internet by assuming it’s a robust proof of something.

This paper is very bad. The numbers in the abstract don’t even add up, which any reviewer should have caught. To be honest this feels like an undergraduate level assignment where students are asking to give a survey and do some statistical analyses. The students usually pick a topic close to their own life (like Super Mario Games) and then come up with some result by playing with their survey numbers until they find something.

[ThePollingStone]

This study reminds me of the types of projects I did when I took statistical psychology classes in undergrad. I was hoping to see data taken directly after participants had actually played the games in a controlled environment. Also, why focus on just Nintendo games?

Judging by the authors' affiliations and Nintendo-approved rhetoric, this does appear to be a shill.

[LeoWattenberg]

> They just surveyed some college students and drew conclusions by running statistical analyses on the data until they got something that seemed significant.

Is this just cynicism or based on anything? From reading the methods section it doesn't appear this is what happened

[Aurornis]

From the paper:

> Methods:

> We used a mixed methods approach. First, qualitative data were collected through 41 exploratory, in-depth interviews (women: n=19, 46.3%; men: n=21, 51.2%; prefer not to disclose sex: n=11, 2.4%; mean age 22.51, SD 1.52 years) with university students who had experience playing Super Mario Bros. or Yoshi. Second, quantitative data were collected in a cross-sectional survey…

So interviews with a biased sample (students with experience playing the game) and then a survey.

Also, try adding up those n= numbers. They don’t sum to 41. The abstract can’t even get basic math or proofreading right.

If the body of the paper describes something different than the abstract, that’s another problem

EDIT: Yes, I know the n=11 was supposed to be an n=1. Having a glaring and easily caught error in the abstract is not a good signal for the quality of a paper. This is on the level of an undergraduate paper-writing exercise, not a scientific study as people are assuming.

[nebezb]

Seems like n=11 should have been n=1. Use 19, 21, and 1 as a numerator of /41 and you end up with all the same percentages written in the abstract. A typo that should have been caught, but surely nothing more than that and certainly not substantive enough to qualify the claim below:

> This paper is very bad. The numbers in the abstract don’t even add up, which any reviewer should have caught.

[Aurornis]

> A typo that should have been caught, but surely nothing more than that and certainly not substantive enough to qualify the claim below:

Such an obvious error should have been caught by the authors proofreading their own work, to be honest. Any reviewer would also catch it when evaluating the quality of the sample size.

I find it strange that people are bending over backward to defend this paper and its obvious flaws and limitations.

[smallerize]

It looks like "prefer not to disclose sex" was typoed and should be 1 instead of 11.

[gs17]

It does seem to be cynicism, they're convinced the authors "gave people surveys with a lot of questions and then tried to find correlations in the data", but nothing indicates they did more than the 9 questions (plus one more for sex as a control) the paper includes, and restricted it to only Mario/Yoshi players. Ten questions is pretty short.

[Aurornis]

> and restricted it to only Mario/Yoshi players.

Do you not see the problem with drawing conclusions from a sample set that pre-selects for Mario/Yoshi players?

How do you think they’re determining that playing Mario/Yoshi prevents burnout if they only surveyed Mario/Yoshi players?

I really don’t understand all of the push to support this paper and disregard critiques as cynicism. The paper is not a serious study, or even a well written paper. Is it a contrarian reflex to deny any observations about a paper that don’t feel positive or agreeable enough?

[gs17]

I've critiqued it plenty in other comments, including that exact issue. However, that doesn't mean they "gave people surveys with a lot of questions" to p-hack, it seems like a study designed (albeit not well designed) to test one specific hypothesis. I see no reason to question that they did the methods as described in the paper, which were designed to test this very specific thing (they didn't even test "childlike wonder" in general, just self-reported Mario-induced childlike wonder), but their conclusions aren't supported by their data. If they were p-hacking as you accuse them of, why not have more questions? Why not survey non-Mario players too so there's a new variable to create significant results out of a null?