[zawerf]

Can someone explain to me what researchers considers "scientific validity" in psychology?

I feel like human behavior will change as culture evolves. For example how people react to a plane hijacking drastically shifted after 9/11. In a similar way the Stanford Prison Experiment probably permanently changed a part of human culture so that it will no longer be replicable (regardless of whether the original setup and results were valid or not).

How do you conduct science when whatever important truth about human behavior you find invalidates the same findings once that knowledge enters popular culture?

[samch93]

In my opinion, if such a culture- and time-dependent phenomenon is interesting, it is still worth studying. If this dependence is obvious, researchers should simply not claim that the phenomenon is generalizable.

The replication crisis has provoked many discussions on similar issues among psychologists, i.e. some dinosaurs question the value of direct replication studies and prefer conceptual replications (see for example this paper by Simons https://www3.nd.edu/~ghaeffel/Value.pdf). I believe that direct replication is the only way to verify the credibility of scientific discoveries, and certainly most new generation psychological researchers agree on this.

[leftyted]

> I feel like human behavior will change as culture evolves.

In some ways it will but in other ways it won't. There are many patterns of human behavior that have not changed, ever, as far as we know.

> How do you conduct science when whatever important truth about human behavior you find invalidates the same findings once that knowledge enters popular culture?

I don't think this is the problem.

I think the problem is that we don't have a theory of the mind and therefore psychology is a field where only erudition is possible. I don't consider psychology, or other "social sciences," scientific. These fields generally lack predictive power and tend to be necessarily qualitative. That's not to say that they're useless but it's an important distinction.

I don't think it's clear that we'll ever have a good theory of the mind. Maybe understanding your own mind is like jumping over your own shadow. How can you understand the thing that allows you to understand?

[abathur]

I think this is exacerbated by a dual blind-spot in both our thinking and languages regarding the difference between "how something was the last time we checked" and "how something intrinsically/durably/permanently is and will be no matter how many times we check". It probably isn't something we've needed in large quantities for long.

Your question focuses on psychology (and I agree it's important here), but this probably happens to greater and lesser degrees when studying any complex adaptive system.

Say you go study the carbon-uptake potential of a few species of tree, model how planting more trees could impact the climate, and click the publish button.

Your research may already be out of date. Your ability to completely control variables in a complex system has limits. Most of the environmental measurements that went into your model are moving targets. Even if your model is predictive at publication, there's a good chance there are unmodeled dependent variables lurking.

What if your research captures the zeitgeist? What if you inspire the planting of a hundred-billion trees? Do your measurements consider lone trees? Does the density and surface area forested (and eventually the weather-shifts a large forest triggers) impact how it grows and functions as a carbon sink? What if your research inspires someone to plant a massive monoculture that ultimately incubates and spreads a parasite that ultimately destroys the species?

[abathur]

To unpack a smidge, I think the first N steps all involve developing the perception and language to spot, tag, and catalog instances of this problem.

There's a fundamental shortfall in the quality of our thinking relative to the task (and the ultimate answer may just be that we're too time-bound to think this way). It's in the study designs, papers, abstracts, and unavoidable in the popular press.

[candiodari]

The reality is that all humanities, not just psychology, do not claim to be value-free. In other words, psychologists (the serious ones) do NOT claim:

1) that patients biases aren't reflected in their research

2) that their own biases aren't reflected in their research

And of course ... there's the https://en.wikipedia.org/wiki/Rosenhan_experiment

[candiodari]

(note that in the Rosenhan experiment, ALL examined psychiatrists had a false positive rate of 100%. Anything significantly over 50%, of course, can only be caused by systematic fraud. That's 12 separate institutions, 20 psychiatrists, in 5 US states. ALL frauds (willing to diagnose patient with an illness despite having no confidence in them actually being ill)

[austincheney]

Peoples response and participation to culture changes as culture changes, but human behavior is largely defined by more primitive physiological and face-to-face interactions that exceed cultural boundaries.

[zaat]

[Daniel Kahneman]... sat down for a long discussion with Sam Harris recently to talk about his life’s work. Harris asked Kahneman if he himself is any different given what he understands about science after five decades of research. Kahneman’s response may surprise you:

> Not at all. In terms of my intuitions being better than they were — no. And furthermore, I have to confess, I’m also very overconfident. Even that I haven’t learned. It’s hard to get rid of those things.

> I’ve been studying that stuff for over 50 years and I don’t think that my intuitions have really significantly improved.

https://awealthofcommonsense.com/2019/03/daniel-kahneman-on-...

[jonnycomputer]

The very traditional view of science as the formulation and testing of universal laws applies fairly well to the study of electro-magnetism, for example. The properties of electro-magnetism, for example, it is reasonably safe to assume that the properties of electro-magnetism are the same today, as they were yesterday, and the properties are the same whether in New York, or in Miami. But this view of science is completely inadequate outside of these, so-called, hard sciences. For example, philosophers of science have tried very unsuccessfully, to rope in evolutionary biology into this view of science. The fundamental problem is that the object of study changes, in space, and in time. If there is a universal law, it is highly contingent--the number of conditions in the if-clause, is enormous. And the most famous "law" in evolutionary biology, the Theory of Natural Selection, is itself entirely empty of physical content: its an algorithm, that can, or may not be, instantiated by a physical system, and because there are many ways in which the algorithm can be instantiated, getting the nitty and the gritty to actually formulate predictions requires the specification of an enormous number of background facts. Instead, evolutionary biology is a model-based science. Scientists evaluate models, models that only partially explain limited portions of the world, and these models usually explicate a process. We have families of models, describing Darwinian processes under different regimes.

I do expect that some experiments conducted in psychology are non-repeatable, especially if they become famous. The question isn't whether it was science, but whether the target phenomena is important enough, and stable enough, to warrant our interest.

In anthropology, Clifford Geertz famously argued that if there are laws of culture, they must be entirely uninteresting ones. His approach, instead, was rich description in place of theorization. His arguments are compelling, however, he too had this notion of science as universal law. I would argue that a model-based notion is much more productive, and much more accommodating to the problem domain. Let me go one step further: what one should focus on instead, is process by which phenomena observed are instantiated. Similar processes can result in quite different outcomes, and similar outcomes can be the result of very different processes.

I work in the field of decision neuroscience. We work, largely, by linking human behavior and neuroimaging to fitted parameters in a family of reinforcement learning models (see: Reinforcement Learning: An Introduction, Sutton and Barto). Decision making processes are largely invariant, even when knowledge, and domain-representations, might change with changes in popular culture: that is, changes in behavior can be accounted for by changing the priors, or input information, into the model.