You just look at the distribution of p values that are used to support the authors' hypotheses. If the distribution is skewed high, then something fishy is going on.
Interesting approach -- get the scientific community to agree on the mathematical principles first, before anyone specifically is outed as cheating.
But this article feels like reading a teaser chapter of a bigger story.
> The amount of (toil) required to actually create data like this from scratch if (very) nightmarish. It’s a task drastically out of reach of the (foolish people) who’d try such a bush league stunt in the first place.
This assumes that all experiments lead to publications. We know there's a strong publication bias, and that the bias favors positive results, and dramatically favors unintuitive positive results. Which means you need to find correlations where none were expected. How many experiments do you need to get a significant correlation when there is none? Hint: more than one.
It's also worth noting that it wouldn't be difficult to produce a genetic algorithm using various statistical checks, including this one, as a fitness function.
Just to address your comment about publication bias – we do have techniques for detecting that across a set of studies on the same subject, namely funnel plots:
I don't agree with the idea that faking data is more difficult than running the experiment. A lot of courses in universities now teach R or something similar to run statistics. Relatively simple "monte carlo" simulations would provide results which satisfy the GRIM test.
A more practical problem with applying this on age data specifically is that everyone I looked at (granted, only a handful of CS papers) only gave one decimal place, not two.
I work together with a psychology professor (was head of department) now and then. She said there's a large problem with students, even graduates, just not understanding statistics and math properly (or at all)
Some of these might be simple errors, with results being typed in from other documents. Authors who are worried about making such errors might want to consider using methods of reproducible research, e.g. writing *blah had mean value `round(mean(x), 3)` (n=`length(x)`)" or similar in Sweave, where the items in the back-ticks are R code working on the actual data. This is a bit more work, but it prevents transcription errors, and also a pernicious type of error that comes about by adjusting the data analysis during the writing process.
Not really...there's a reason while psychology is not considered science by non-psychology scientists: it's not because the researches couldn't be done in a scientifically useful way, but because even the teachers don't care about math. I think Facebook and Google and other ad agencies have a better model about human behavior than psychologists (although they only have data on a specific part of human behavior)
I've done a similar trick where the ratio of two secret integers is released publicly with many significant digits and you can sometimes find the two integers by brute forcing the division over all possible values. Does anyone know a name for this approach?
http://www.p-curve.com/
You just look at the distribution of p values that are used to support the authors' hypotheses. If the distribution is skewed high, then something fishy is going on.