[gerbilly]

I've sat this out for long time, but I've got to say that it's hilarious watching the HN community—mostly made up of engineers—constantly harp about the replication crisis in Psychology.

In some cases effect may be real even though the studies aren't replicating today. There are several factors.

Replication in psychology isn't as easy as in the physical sciences.

Let me use an example from another field to give you an idea. If I asked you to translate the sentence: "He took the train for London at eight am." into French, would you translate "London" as "London" or as "Paris?"

In other words the social context—social norms, expectations, attitudes—from when many of these studies which don't replicate today has shifted, such that re-running the study verbatim might fail to...replicate.

However say you do try to adapt the study to adjust for the shift in context, then you could also say that the replication isn't valid because you didn't re-run the study exactly as specified.

People aren't robots, so we can just re-run the unit tests on them and expect the exact same results. People aren't even self contained robots, we are social beings, and are very strongly affected by our peers. We aren't swimming in the same water that we were swimming in when these studies were done.

We need new studies for each of these effects that are designed with today's reality in mind. But. No matter how one tries to 'replicate' an old psychology result, it will leave room for skepticism. It just won't be able to account for the 'translation' required without leaving room for doubt.

I guess the most important thing we could learn from this is that it's important to replicate any current studies right now, and not wait forty years to do so.

[pas]

No, the problem is a lot deeper than just "capturing the protocol in the methods section is hard". (There are very interesting articles about how word frequency shifting might be a confounding factor, which can cause old - from the 70s and 80s - papers to fail replication. But then there's no follow up with a new corpus and new replication.)

It's not just bad (vacuous) science, at its core it's people who act in a bad (selfish, non-scientific) ways: https://statmodeling.stat.columbia.edu/2016/09/21/what-has-h...

> I guess the most important thing we could learn from this is that it's important to replicate any current studies right now, and not wait forty years to do so.

Yes, that too, but what's even more important is to shift into a mindset that starts with good models, good data generation processes (ie. experiments), then we can check and compare their predictive power. Otherwise we get these statistically flawless abominations that prove ESP:

https://statmodeling.stat.columbia.edu/2011/01/11/one_more_t...

And, sure, yeah, it's hard to do this. But otherwise we'll have nothing more than just-so stories supported by random data that happened to break through some significance threshold.

[deskglass]

The replication crisis isn't merely in psychology, old studies, or studies that didn't carefully record their methodology. Eg the review I linked mentions a study from 2006 that didnt replicate in a larger trial. The one about stories involving immorality priming people to use more antiseptic wipes. It's weird to me that someone would write a book and reference that study without mentioning this replication failure. Especially in the context of the many other such failures to replicate for priming.

Theories ought to be backed by experiments that replicate. Maybe describing methodology precisely is hard, but replication is how we know a phenomenon is actually real and not p-hacked or otherwise mistaken (eg by experimenters mis-measuring).

[slingnow]

So what you're saying is all study designs are acceptable. All study results should be trusted. We should trust studies that can't be replicated, because "something about translating London into Paris".

What you're describing isn't science. If you want psychology to be held to a different standard, then that's fine. But then you can't call it science.

[gerbilly]

Thanks for your ungenerous and bad faith summary of what I wrote.