[feoren]

I think there's a common pattern on Hacker News that goes something like:

A: Overly broad generalization of a huge body of work put together over 100 years by tens of thousands of professionals

B: Ugh, hate this take from armchair experts

A: Okay, then give me all the examples! Otherwise you're proving me right!

I happen to think your overly broad generalization is more right than wrong, but I also recognize the silliness of asking to be "enlightened" on an entire branch of human endeavor via internet comments. This is a problematic argument form, and someone calling out this behavior does not prove you right.

So let's be clear about what "enlightening you" means. If your argument is "psychology is based on a fundamentally flawed/useless study design (surveys) and we can never learn anything real from it", then a few examples of reproducible, interesting, not a-priori obvious results from surveys should be sufficient to show that we actually can learn real things from surveys. (And be careful not to fall into the "I could have told you that!" fallacy.) Luckily, this question was already asked on Reddit, and I think there are some strong examples:

https://www.reddit.com/r/AcademicPsychology/comments/qktt6h/...

On the other hand, the field is absolutely rife with problematic study design and even some entire psychology departments (e.g. Stanford) seem to be completely rotten. The most salient example of this is the "implicit bias" studies that came out of Stanford. Their study design was something like:

Task 1: Associate good words with white/Christian/American themes as fast as you can

Task 2: Associate bad words with "foreign" themes

Task 3: Associate good words with white/Christian/American themes again

Task 4: Associate good words with "foreign" themes

And the result is: you're racist because Task 4 takes you a few milliseconds longer. It never occurred to them (or it did and they intentionally forced the result) that in Task 4, you're literally unlearning what you've just practiced 3 times. It was one of the most blatantly bad studies I've ever seen in my life and I didn't see anyone else calling out how problematic it was, because Stanford.

So in general I actually agree with your take: the field is rife with junk science, some of it obvious, and almost certainly some of it intentional. But please also recognize that "I'm an expert in tech and therefore everything, and if you can't prove me wrong in an internet comment then that proves me right!" is a very problematic argument style. It sounds like you're trying to prove yourself right, and a much more efficient way to get smarter is to habitually try to prove yourself wrong.

[cheeseomlit]

I appreciate the productive answer- You're right, re-reading it now my tone was more argumentative than inquisitive- Itd be foolish to dismiss such a large body of work as 'useless' and I hope it didn't come off that way- Of course understanding human psychology is immensely useful for all sorts of reasons

[SubiculumCode]

To be fair, most studies of implicit bias are randomly ordered on a trial to trial basis.

[mike_hearn]

I agree with the overall thrust of your argument, but:

> be careful not to fall into the "I could have told you that!" fallacy.

That's not considered to be one of the standard logical fallacies as far as I know. Why would it be fallacious? Social studies are rife with findings that are either extremely obvious to everyone, extremely obvious to conservatives specifically (because psychologists are nearly always on the left), or extremely obvious to anyone who reads the study design.

I recently wrote an essay about why replication studies can't fix science [1] and one of the problems cited is the prevalence of studies that aren't worth replicating because "I could have told you that". Examples include silly papers like [2], which is literally titled "People's clothing behaviour changes according to external weather and indoor environment" yet somehow manages to also say, "It is evident that further studies are needed in this field", or [3] saying that the average male student would like to be more muscular.

But there are less silly examples which crop up due to the ideological bias in the field. Academics purge any conservatives they find, meaning that social studies spends a lot of time and money investigating things that are considered obvious outside of far left spaces. Jonathan Haidt is famous for arguing that this is a problem (albeit, not actually doing anything about it). As an example highly apropos to this thread, psychologists recently started discovering that stereotypes are usually accurate. Much other work in psychology is built on the suspiciously circular premise that stereotypes are either fictional and thus mere folk intuitions, as Mastroianni would put it, or are accurate only because people believe they are accurate (the field of "stereotype threat" is like this). On the left the idea that stereotypical achievement gaps are socially constructed is considered obvious and a matter of faith, to people on the right the opposite is true: idea that they reflect actual truths about reality is the obvious idea.

So even if you set aside the offensively wasteful, there's still a lot of scope for study claims to be considered obvious by some and not by others.

[1] https://blog.plan99.net/replication-studies-cant-fix-science...

[2] https://www.sciencedirect.com/science/article/abs/pii/S03601...

[3] https://openurl.ebsco.com/EPDB%3Agcd%3A4%3A12322516/detailv2...

[feoren]

> That's not considered to be one of the standard logical fallacies as far as I know

I don't really care whether it's on The Official List of Logical Fallacies™ or not, and in fact caring too much about this list is itself a bit of a fallacy. Nor do I necessarily consider "I could have told you that!" necessarily a logical fallacy; more like an emotional fallacy. (But humans use emotions when trying to understand things!) I consider a fallacy to be something which is an "attractive but wrong" step in an argument or rationale. There are two reasons why I consider it a "fallacy" by that definition:

1. Humans dramatically over-estimate the obviousness of an idea after they've already heard it. Once you know the answer, you basically become immediately unable to estimate how obvious that answer was beforehand. This is highly evident in mathematical proofs, e.g. when "obviously right" things turn out to be wrong, often with an "obvious counterexample". Both sure feel completely obvious, depending on what you know!

2. Even obvious things are worth testing. Plenty of things that seemed obvious have turned out to be wrong. This is not evidence of wasted funding. Obvious things can also be related to less obvious things. Your 3rd example shows this: there's a 2nd related hypothesis they're testing: "men would believe that women would find a more muscular shape more attractive than women actually report". So men tend to want to be more muscular, and also men tend to think women find high muscularity more attractive than they actually do (or they think women's "ideal muscularity" is higher than it actually is). This may be an example of "overturning our intuitions" whose understanding could improve outcomes -- if it replicates, anyway. Hardly an example of a pointless study.

That being said, humans actually do seem pretty good at being able to "tell you that" beforehand. There are some fun quizzes you can take [1, 2] to see if a study replicates beforehand, and you'll probably do pretty well on them. But that doesn't necessarily mean that we shouldn't do the tests anyway! We can still be surprised.

> Academics purge any conservatives they find

This is extremely dependent on the department and institution, and in general is way overblown. Yes, this can be a problem in some departments in some of the social sciences. On the other hand, in economics departments, it can be liberal ideas that are verboten. There are fads and politics everywhere, but mostly science is doing OK at considering ideas on their actual merits -- eventually. Meanwhile a lot of people lob angry criticisms at academia for rejecting their bad ideas because they're bad and wrong, assuming

> psychologists recently started discovering that stereotypes are usually accurate

I suspect this is way too strongly worded for what's actually being found (and replicated). Citations would be very much appreciated. Be wary that slight changes in the mean of a population can be detected in tests with p < whatever epsilon you, and also have oversized effects on the tail ends (e.g. professional sports), but give you almost no predictive power for individuals you meet on the street. If "stereotypes are usually accurate" means "population X is slightly more likely to do Y with near-certainty", that does not mean "most X do Y" nor even "a random member of X is significantly more likely to do Y than a random member of not-X". One of the reasons these kinds of studies are considered problematic is in how easily they can be misconstrued to justify racism.

[1] https://mru.org/teacher-resources/active-learning/will-it-re...

[2] https://80000hours.org/psychology-replication-quiz/

[mike_hearn]

Sure, here you go. This is the first I found with a few seconds of searching so I don't claim it's the best citation but it gives an overview of the research on stereotype accuracy:

https://spsp.org/news-center/character-context-blog/stereoty...

Note the reference to over 50 studies, and stereotype accuracy being amongst the most replicable of findings in social psychology. Not very surprising given they're literally asking "are things most people believe to be true actually true" - this is a question that's going to obviously yield a lot of big effect sizes and high levels of replicability, but it's also trivial by definition.

> in economics departments, it can be liberal ideas that are verboten

That seems doubtful, depending on how you define "liberal". Academic economics still has a notable left leaning bent. If we check out the last two issues of QJE, one of the best known economics journals, we see a large number of papers on typical liberal fascinations that have little to do with conventional economics, things like gender pay gaps, domestic violence against women, how to socially engineer people into taking vaccines etc:

https://academic.oup.com/qje/issue/139/3

https://academic.oup.com/qje/issue/139/4

Also, the basic premise of academic economics is that you can treat the economy as something knowable and controllable from the outside, whereas a more libertarian take would be that the economy is the process of everyone collectively figuring out what's both true and desirable, thus you cannot "step outside" the system in any meaningful way by definition. So the very nature of studying economics in a university is to some extent a left wing starting point.