[logicchains]

>When you think about it, it's insane that metascience isn't studied more. We're going to question every little detail about the universe, but we're just going to take on faith that peer-review and publication in journals is an effective method for weeding out "bad" science?

In a sense we do have this: engineering and finance. Engineering turns good hard science into new tools, machines and weapons, and Finance turns good (predictive) soft science into new ways to make money.

[jhap]

> In a sense we do have this: engineering and finance. Engineering turns good hard science into new tools, machines and weapons, and Finance turns good (predictive) soft science into new ways to make money.

I think this is a common critique, but I also think it is missing the point. What if the question of interest isn't so easily verifiable like in Engineering? Do we just throw up our hands and give up on those questions? [The alternative to good social science is not no social science, it’s bad social science](https://statmodeling.stat.columbia.edu/2021/03/12/the-social...).

Finance is also a bit tautological in this regard. It seems that often prediction models are impossible to disprove (e.g., our arbitrage method doesn't work anymore, the market updated). Yes good for putting skin in the game, but doesn't seem like it does much to advance our long-term understanding of humans.

[logicchains]

>What if the question of interest isn't so easily verifiable like in Engineering? Do we just throw up our hands and give up on those questions? [The alternative to good social science is not no social science, it’s bad social science](https://statmodeling.stat.columbia.edu/2021/03/12/the-social...).

Some things may well be complex enough that it's simply impossible, with the amount of resources available to the average university, to conduct a thorough enough study on a representative enough sample that accounts for enough confounding factors to make a statistically sound prediction that generalises. If this were the case for a significant proportion of the subjects of study of a particular field, then it might well be better to "give up" and admit we don't and cannot know, otherwise we're essentially creating a factory for bad science (as the available resources relative to the scope of the problem aren't sufficient to create good science, and there's no negative feedback to stop the bad science).

[ImprobableTruth]

If there's "good social science" that can't be used to make predictions, what differentiates it from bad social science?

[akiselev]

Its utility [1]. The social sciences study a lot of things that people in group A intuitively understand that group B can be completely ignorant of - say, for example, how to navigate a complex social structure like office politics in a modern workplace. Making any sort of predictions about intangible outcomes where the Hawthorne effect is in full effect is pretty much impossible since group A will respond to the new knowledge gained by group B, in effect changing the system we're trying to predict. Individual's psychologies respond to the changing psychology of the group in nondeterministic ways (at least, relative to our ability to collect data on input variables and internal state).

We can bikeshed what makes something a "science" till the cows come home but the philosophy of science and epistemology were not settled with Bacon and Popper - the end goal has always understanding in the broadest sense. Those studies have value as long as they help someone make sense of and adapt to the social systems they're in. It does mean though that those studies should be approached with extreme caution (see the decades wasted on string theory) and anyone basing their research off past results needs to carefully validate their assumptions.

[1] I think in this case "predictive" as a scientific term of art is too restricting. Social sciences often deal with very personal interactions that appear nondeterministic at the scale of a society but are relatively predictable when applied to a stereotypical office or school setting.

[ImprobableTruth]

I don't understand what difference you're trying to make between utility and predictive power. If you can give information on what approach in general will be better to approach office politics that is just a prediction. It doesn't mean that these predictions have to be always right, but if they don't have predictive power and are no better than a coinflip, that "understanding" is just a post-rationalization that doesn't provide any utility at all.

At the very least, it seems to me like the person I originally responded to would also disagree with judging social sciences for its "utility" - the article they linked specifically contrasted it with the natural sciences that "solve problems".

[akiselev]

For something to be predictive in a scientific sense, it has to be repeatable. There is so much variety in individuals and their environments that most social sciences have little repeatability - they mostly study affluent western college students who have time to volunteer for college psychology studies. However, if you're mostly an affluent western college student, chances are that you can take some value out of the studies because they're selected for your environment rather than humanity as a whole (which is what they purport to do by claiming to study 'psychology' rather than western college students specifically).

Closest analogy off the top of my head is psychiatric drugs: their efficacy is generally bottom of the barrel except for some group with factor X (each drug has their own unique factor X). For the vast majority of these drugs, we have no method of screening for whether a person has factor X - we don't even know what it is most of the time - so doctors have to go through a process of trial and error with patients until they find the right drug or combination. Once they do, it's like a night and day difference for the patient, yet if we applied the same standard of evidence for psychiatric drugs that we do for blood pressure pills, we'd never make any progress. A lot of the drugs look like they don't work in phase III and we have no way to predict which drug which help which patient but the patients figure it out with their doctors because they have actionable data, even if it isn't predictive in general.

[hpoe]

I think this is an important point, ultimately good science will produce, verifiable, testable, actionable results. Until you have that no matter how much math you use, how many lab coats you've got, no matter how many journals you publish in you're just sitting there playing with strings.

[jrochkind1]

Telling when this has happened can be less trivial than you would assume though. People (scientists even) were sure that phrenology produced verifiable, testable, actionable results for a generation or two.

In the long run it usually comes out, but the run can be longer than you think, and you may not be where you think in it with regard to any particular current theory. I wonder what things we know all "know" are proven by science will be dismissed by later generations. (I personally guess a lot of genetics-related stuff will be).

(Note that something doesn't need to be verifiable, reliable, or true to be "actionable". You can act on anything...)