[m12k]

When you think about it, it's insane that metascience isn't studied more. We're going to question every little detail about the universe, but we're just going to take on faith that peer-review and publication in journals is an effective method for weeding out "bad" science?

[munificent]

> it's insane that metascience isn't studied more.

It is, probably more than is apparent. See:

https://en.wikipedia.org/wiki/Philosophy_of_science

https://en.wikipedia.org/wiki/Sociology_of_scientific_knowle...

There is an interesting recursive problem here, though: what tools do you use to scientifically analyze the scientific process? Whatever tools you use will themselves be hobbled by the same systemic flaws you are trying to understand.

Also, as any sociologist will be happy to tell you, incentive structures and other human group behavior gets in the way. It's probably hard to get funding for a study that shows that all the other departments at your university aren't quite the flawless seekers of truth they appear to be.

[analog31]

Indeed, and if you think science is bad, imagine metascience.

[fnord123]

When you think about it, it's insane that metametascience isn't studied more. We're going to question every little detail about the scientific process, but we're just going to take on faith that peer-review and publication in journals is an effective method for weeding out "bad" metascience?

[caenn]

Indeed, and if you think metascience is bad, imagine meta-metascience.

[svieira]

Meta-metascience has been studied _extensively_ by previous generations, where what this generation calls "science" was called "natural philosophy" and interesting discourse and study was had on such things as:

* The study of knowledge (epistemology): https://en.wikipedia.org/wiki/Epistemology

* The study of existence and what exists (ontology): https://en.wikipedia.org/wiki/Ontology

* The study of the purpose of things (teleology): https://en.wikipedia.org/wiki/Teleology

It turns out that these are hard areas of study and it requires a lot of properly focused leisure time to understand properly. Most people don't consider these things, wing it, and wind up working with a half-baked meta-meta science of their own creation ... oh, I see what you mean :-D

[analog31]

Where's tail call optimization when you need it?

[jonsen]

Would be nice if tail call optimization solved the halting problem.

[IIAOPSW]

You've caught an Orobus by the tail!

[nicoburns]

I might be biased as someone who studied philosophy. But I wish Philosophy of Science was mandatory in more science degrees. A lot of scientists don't seem to be familiar.

[andi999]

Can you point to something good and useful in the Philosophie of Science? I mean let's say physics is as good as it gets (laws in mathematical language, strong experimental evidence, solid results); from my perspective a success. But it has nothing to do with how like popper imagines science. So insisting on like a popper process would also not be benificial for physics in my opinion. But probably there is better stuff, I just couldn't find it. (with a popper mindset nobody would understand why we still study 'falsified' theories like electrodynamics)

[nicoburns]

Some key things for me would be the ideas that:

1. There is no such thing as objectivity in inquiry. There is always a scientist interpreting things, and they are always looking at things through the lens of their own biases and cultural norms. The best you can hope to do is to be aware of your limitations.

1. Statistical evidence on its own is not always a good basis for believing something to be true. Typically you also want a good theoretical model, and an understanding of the mechanism that underlies the observed phenomena (physics is good at this, psychology not so much).

3. That there is more knowledge than is detectable through statistical methods (currently at least). And that lack of evidence from statistical studies does not necessarily constitute good evidence that a theory is false.

[LudwigNagasena]

1. This is pretty trivial. Why do you think most scientists aren’t aware of their limitations?

2. This is studied in statistics classes. It is impossible to analyze data without a statistical model, so of course your conclusions rely on it. So this is also quite trivial.

3. Again, are scientists oblivious to this simple notion? I doubt it.

[twhb]

Why is it that mirrors flip your image left to right, but not top to bottom?

That’s a question that Richard Feynman supposedly asked his grad students. Once you answer it, I think you’ll realize that #1 is anything but trivial.

Answer below, stop reading this comment if you want to figure it out on your own.

It’s because you’re comparing the mirror image to what you’d look like if you walked around the mirror, instead of to what you’d look like if you floated over the top. This assumption of horizontal travel is incredibly deeply engrained in humans, to the point that the English language doesn’t even have up/down equivalents to the words “left” and “right”, i.e. a word that means the direction closer to your head than to your feet regardless of your orientation.

[sprafa]

A lot of this stuff applies more to bad scientists, of which there are many.

[dennis_jeeves]

LudwigNagasena, to you it may be obvious ( as it was to me). However I discovered ( much to my disappointment) that it is not obvious to career 'scientists'. A fool with a phd is still a fool. What used to the domain of primarily religions (rituals, superstitions etc.) is now the domain of several offshoots, one of which is mainstream science.

[bsder]

> Can you point to something good and useful in the Philosophie of Science?

Every scientist should be able to answer the particular tenets that they take on faith.

Scientists do take on faith that the universe is causal and that the rules today are the same as the rules yesterday.

Yes, scientists double check these assumptions over and over, but they can never "prove" them.

This kind of introspection is important for science to set itself apart from religion, for example.

[andi999]

I meant more like book, or a philosopher.

[missingrib]

Yeah I think the fundamental distinction is that metascience is probably at its core going to be philosophy, but that doesn't have to make it any less rigorous. And I agree, Philsoophy of Science being included in more science degrees would be great. One of the most interesting courses I took in my undergrad.

[SiempreViernes]

It's not that useful in practice though, for many fields a bit more ethics is a much more pressing issue.

[chromaton]

>what tools do you use to scientifically analyze the scientific process?

Iterating on existing systems to see if you can get results to converge and also testing new systems to see if they also result in known good values.

[munificent]

It's fun to realize how deeply ingrained the current scientific process is in our way of thinking.

All of these ideas that you tacitly take for granted are itself mutable parts of the scientific process:

* That iterative improvement and hill-climbing is an effective process for improving results.

* That replication of experiments and convergence is a truth-generating enterprise.

* That truth can be expressed numerically.

* That there are some values that are "known good". By what process? According to whom?

To be clear, I don't disagree with those. However, these rules aren't baked into the firmament of the universe. They are processes we humans have chosen to apply in our social process of reaching conensus on truth. In other words, this list here isn't physics, it's technology.

It's entirely possible to imagine a culture whose truth finding bodies don't take for granted one or more of these rules at all. That culture might be more or less effective (again, according to what metrics?), but it would still be well-defined.

[dragonwriter]

> That iterative improvement and hill-climbing is an effective process for improving results.

Not essential to science; in fact, there’d a major viewpoint within metascience that explicitly rejects this as popular mythology of how science works in practice, holding that models change by revolution more than evolution.

> That replication of experiments and convergence is a truth-generating enterprise.

Not part of science, in the same way that your use of “truth” later is not.

> That truth can be expressed numerically

No. While scientism may make essential connections between science and truth, science itself only depends on useful predictive models being expressable, not about truth being expressable, numerically or otherwise, or even being a coherent, meaningful concept.

> That there are some values that are "known good"

That's not only not essential to the scientific process, but contrary to something that is: that all results are contingent.

[chromaton]

Yeah, I'm reminded of Zen and the Art of Motorcycle maintenance where the author was driven to insanity by the quest to define "quality".

[didibus]

> That culture might be more or less effective (again, according to what metrics?)

Isn't this the idea of "free will". That you get to choose for yourself the metrics you want to optimize your life for?

Now I think that you'll use a combination of learned and inherited desires for it. But the idea here is that each individual can express those desires, and then the "success" of a society is thus to maximize each individuals success, even when they differ in their metrics.

That's a made up concept as well, but I think it still stems from individual desires. We've just mostly all individually observed that an organized society that compromises with each other to maximize each and everyone's individual desires has less risk to our own desires being squandered.

The alternative would be to try to achieve power over others to maximize your desires, and maybe from history and life experience, people have found that to be not sustainable or only achievable for a few, thus your chances at it are lower.

In essence, I think I'm saying that it seems over time people know their desires, but don't know how to beat fulfill them, and this is the metric.

[svieira]

> That truth can be expressed numerically

I don't think this is what is held by those-who-do-science-and-philosophy-at-large (though it may be a generally accepted hand-wave, I don't know). See, for example Category Theory for a branch of what-I-believe-would-generally-be-called-science that doesn't use numbers, but instead expresses things with sets and relations.

The logician is the intersection of the set of all scientists and the set of all philosophers.

[panagathon]

I think I disagree, since mathematics cannot be reduced to logic (or any other instrumentalisation).

https://plato.stanford.edu/entries/philosophy-mathematics/#M...

[lupire]

That article says that mathematics is ill-founded, not that math has something inexpressible by logic. Mathematics can be reduced to logic that is ill-founded.

[xxpor]

The only nit I'd pick with the list is:

* That truth can be expressed numerically.

Isn't the basic point of quantum physics that this isn't true? We can only make guesses with probabilities, but we can't know the actual truth, and therefore can't express it numerically.

[munificent]

Sure, but the probabilities are numbers too. Which is again sort of acknowledging the need to fit quantum mechanics into a numeric framework.

Imagine you were studying ice cream flavors. You might design a study like, "We'll ask a lot of people and the flavor that the most people prefer is the best." In other words, the metaprocess you use to design your experiment itself tacitly assumes you need a numeric result. The presumption of comparison and quantifying frames the questions you even think to ask.

But you can imagine an alternate culture that when studying ice cream flavors doesn't even ask questions with numeric answers. It could be, "We'll ask a lot of people to try flavors and write poems about the experience."

We wouldn't even call this "science". Because there is a hidden border around even the term that affects how we are able to evolve the scientific process.

[klyrs]

Funny thing about quantum mechanics... we use numbers to describe probabilities and functions to describe probability distributions. But the mathematical models we use are incomplete (and, looking at neutron decay, incompatible) so can we really conclude that numbers are the right abstraction?

And no, not poems. Elements of finite groups are not numbers, but they crop up in physics frequently. Topologies are not numbers, but they're also significant in physics.

[licebmi__at__]

That seems broad enough to describe the human history isn’t it? So, “keep doing what you do”?

[chromaton]

Not necessarily. You could just fall back on authority, superstition, or navel-gazing.

[dpe82]

That would probably fall under "testing [new/other] systems to see if they also result in known good values."

[fallingknife]

> what tools do you use to scientifically analyze the scientific process?

Engineering. If you can build something that works based on the rules theorized by scientists, they are on to something. e.g. building a skyscraper proves we know the properties of steel to a pretty good margin of error.

[BurningFrog]

For many sciences, like psychology, that's not normally an option.

[sprafa]

Psychology is basically applied neuroscience. They are both in their infancy and we are using the best information we can, but I wouldn't call the body of work comparable to other sciences.

[thu2111]

Which raises the question why we study this subject at all, if it's not actually applicable to anything beyond making viral TED talks.

[rijoja]

Yeah so it is not science but rather philosophy and it is called Epistemology and not metascience. Even though I suppose metascience would be an apt description in a way.

[bluetomcat]

"People" sciences like sociology, psychology and economics can make incredibly misleading claims because one experiment over a small sample of people at a certain moment in time might seem to support a claim, while the actual reason for the observed results is a factor which is never taken in consideration. On the other hand, conducting those experiments over wider demographics and in different points in time means that the study wants to build "universal" models of how each single person in the whole world acts, which is utterly dismissive of the specific local environment around people.

Sociology in particular should always be approached highly critically, because applying those theories and reasoning in its terms often means mass control over people's free will.

[cwkoss]

I majored in psychology in undergrad. A big part of why I didn't look for a psychology focused job is that the science is all so loose. I'd often learn about two different study-backed phenomena is two different classes that somewhat contradicted each other. Or I'd learn in a subsequent class that a previously taught study has been invalidated in one way or another. Almost everything is measured subjectively, so huge parts of our knowledge of psychology are a house of cards resting on assumptions that the diagnostic questionnaires used to measure are accurate and reliable. Many of the measured effects are small, and so it's hard to trust that randomization and controls are sufficient. Replication of results is a major issue.

It all just feels so 'loose' compared to the physical sciences.

[lupire]

The important problems are hard. Avoiding psychology because it's messy is like the metaphor of only searching for your lost keys under a lamppost.

[bingidingi]

Think about how loose medical science used to be (and for how long)! leeches, bloodletting, miasma, ridiculous enemas and all sorts of outright nonsense. We've got a lot more mistakes to make, but social sciences will improve too.

[ch4s3]

To be fair, no one was using statistics and the scientific method to support bloodletting, or miasma theory.

[Karrot_Kream]

No, but let's not forget that much of statistics was originally invented to provide a rigorous underpinning for eugenics. Pearson, who invented many of the most commonly used statistical results, was a prominent eugenicist and contributed greatly to its ideas.

[ch4s3]

That doesn't mean that statistics aren't a useful tool for checking the strength of evidence, or building a case for a hypothesis.

[hpoe]

Honestly all sociology I have seen or been exposed to, including in college, seems to be more interested in acting as a platform to push specific ideas, rather than an attempt to find truth.

Beyond that those involved in sociology seem to believe that a study is the same thing as an experiment and like to believe that constitutes proof.

Ultimately we can't really run AB experiments on society at large because we are living in; however humanity has at its disposal all of history as a case study. My point is if you really want to understand how societies interact and form, and react, and live ask a historian, not a sociologist.

I also would apply most of these comments to economics except there seems to be more diversity of viewpoints, and studies are used less than math to try and provide a veneer of respectability.

EDIT:

If someone feels that history is inferior to sociology for understanding how societies act and behave please tell me why. I want to understand where I am wrong. But I see a lot of our arguments that we are having in society nowadays the same as one's had a thousand years ago, the discussions over Social Media are basically the exact same ones people had over the printing press in Europe, I recently read "The Republic" and there were the exact same arguments I see repeated here.

So if you feel contrary please tell me why, I admit I could be wrong, but want to understand where my reasoning is flawed.

[dash2]

I think you are naive about history.

I'm an economist. If I threw away the half of the data that didn't support my findings, and got caught, I'd lose my job and never publish again. I'm pretty sure the same is true in other social sciences, such as psychology. This is true irrespective of the well-documented problems that the article describes, which certainly also apply in economics and elsewhere, to varying degrees.

By contrast, when historians are caught cutting sentences in half to prove their point, they don't lose their jobs. They don't even lose their Pulitzers: https://davidhughjones.blogspot.com/2020/07/can-we-trust-his...

[peoplefromibiza]

But let's not pretend that historians influence the policy makers as much as sociologists and economists do either.

The damages when they are wrong are orders of magnitude bigger.

They are assumed to be right, sometimes even without proof, until they are tragically proven wrong.

And nobody lose their job anyway.

Have you ever seen a sociologist lose the job because proposed something to a politician that resulted in lots of people having their life ruined?

I never did, honestly.

Have the last three more recent economic and social crisis been caused by historians mistakes?

https://familyinequality.wordpress.com/2017/10/02/sociologys...

[dash2]

I think you underestimate how influential historians are in the long run, by changing how we see ourselves. But in any case, my point was about which disciplines we can trust, not about which are more or less powerful.

[LudwigNagasena]

Economic policy is implemented by the parliament and the executive branch. It rarely closely follows advice by the economists. Even the Fed chair is a lawyer!

[whakim]

The thing about textual evidence is that you can't cite an entire text (obviously). You have to selectively choose what to quote in order to support your claims. Additionally, people can write one thing, and then write other contradictory things. Or they can act in ways that contradict what they write. It is from this totality of evidence that non-quantitative methods draw their conclusions. To get to the point, I'm not necessarily claiming that Nancy Maclean (the historian "caught cutting sentences in half") is in the right here, but if you actually follow the debate it seems quite nuanced and the internet critic hadn't actually even read most of the book they were criticizing (and also clearly has certain political leanings to boot). Certainly nothing like "throwing away half the data that didn't support my findings."

[dash2]

The JEL review which I quote in the linked blog certainly had read the book, and called it "replete with significantly flawed arguments, misplaced citations, and dubious conjectures". And if cutting sentences in half, to remove something which directly contradicts your thesis, doesn't count as historical malpractice, then what would?

[whakim]

It's just not that simple. I'm not making a value judgment on the book (I haven't read it), but a person can say two things in the same sentence and the broader context can make it clear that they're just covering their ass, for example. Perhaps that's not what's going on here. Perhaps the book does constitute "malpractice." But...I think the situation is more complex than you're giving it credit for, and I wouldn't be comfortable drawing conclusions without a greater familiarity with the book and the responses to it. I also don't give a lot of credence to the blog you linked, since they use (as one of their two pieces of evidence) a critique which openly admits it hasn't actually read the thing that is being critiqued.

To your last point, plagiarism, for example, definitely counts as malpractice and humanities professors lose their jobs for it.

[hpoe]

After following up on your sources, I surrender my position. It appears that the quest for truth has largely been abandoned in academia, and that integrity is a fools dream.

We truly are as T.S Elot said the hollow men.

[dash2]

Let's not go overboard now....

[dvfjsdhgfv]

I don't disagree with you, but frankly would be a bit frustrating to limit one's studies of human behavior to just history without trying to understand the dynamics of current societies, trying to understand how they respond to change and so on. Both fields have a completely different set of instruments and very limited overlap.

[jackcosgrove]

The best social science studies involve often accidental experiments, where good experimental conditions occur not because of design but because of happenstance. The analysis of these situations could be construed as a historical case study, or it could be construed as an experiment. I agree that seeing analogues in past societies is not the best approach, but studying history can sometimes reveal experiment-like conditions.

[Macha]

Another similar issue is with data from situations that would be clearly unethical to intentionally create. Behaviour of plane crash survivors standard on mountainsides, castaways, feral children, etc

[notsureaboutpg]

I felt this way as well. But you might benefit from reading more old school sociology books.

C. Wright Mills The Sociological Imagination is great (should have been taught in college to you). Thorstein Veblen's Theory of the Leisure Class is good as well. These really seemed to me like attempts to approach truth, and perhaps that's because of the time they were written in vs the time we live in now.

[rafael_c]

"On the other hand, conducting those experiments over wider demographics and in different points in time means that the study wants to build "universal" models of how each single person in the whole world acts, which is utterly dismissive of the specific local environment around people."

I don't think building "universal models" or observing recurring patterns through analysis of 'experiments over wider demographics and in different points in time' require the ambition to predict a single individual behavior or actions as a corollary.

The problem lies - like you said - with the policymaker. And well more generally with people who extrapolate the results of a paper inadequately.

[skybrian]

The problem is even more pervasive than that. There is an irresistible tendency to try to make universal statements rather than just sharing anecdotes and not generalizing from them.

Like, for example, I just made two universal statements, didn’t I?

[rijoja]

Yeah so you have to make a difference between empirical science and science here really. Which Max Weber who was one of the pillars of social sciences stated around one hundred years ago.

"As such, he was a key proponent of methodological anti-positivism, arguing for the study of social action through interpretive (rather than empiricist) methods, based on understanding the purpose and meanings that individuals attach to their own actions."

https://en.wikipedia.org/wiki/Max_Weber

edit: This is then further developed by the so called Frankfurt School as Critical Theory.

[acituan]

Alternatively, anti-positivist endeavors should find themselves another space to occupy and not piggy-back on an institutional adjacency to actual sciences to posture credibility, authority, attain public funding etc.

[rijoja]

On the other hand one might also argue that positivism is just a school of philosophy and that positivists are piggy backing on a thousands of years old tradition of philosophy.

[bgandrew]

>> thousands of years old tradition of philosophy

imagine those proponents of 'thousands of years old tradition of philosophy' would try to accomplish anything with it. how funny that would be..

[pacbard]

I saw a chart once that made the point that most research studies can be classified along two axes: rigorousness of methods and popularity of results. Most published papers have either high rigor/low popularity and low rigor/high popularity. The trade-off is in the fact that highly rigorous studies only allow for narrow, unexciting results, while popular studies with flashy results will have to compromise on their rigorousness. This is not really a rule, but it is an interesting way to see research and the editorial/peer-review process.

The example discussed in OP seems to fall in the category of low rigor/high popularity. I am not 100% on my history of psych research, but it seems to me that the stereotype threat was all the rage in the late 90s following the publication of Steele and Aronson (1995). OP study seems to follow a similar experimental setup as S&A with a new group of people (Asian-American women).

As far as meta-science is concerned, I think that it remains mostly a part of philosophy (as in epistemology) and the focus of a few (senior?) scholars in each field. There is really no space to publish meta-scientific papers that "shake up" the field and call out established researchers, as editors that publish those pieces could come under similar criticism for their work. I think that it is not an accident that the discussion of the replication crisis in psychology started from blog posts and other non-academic avenues and then found its way to more "established" publications in the field (again, if I remember the context of those conversations).

I really wish that the review process was open. It would be interesting to see the reviewers comments to this specific paper and how the editor decided to pick up and engage with them. All those conversations are usually locked up in some editorial management system and are seldom made public. I don't know if we can really have open science without having open peer reviews.

[derbOac]

I agree with you mostly, but it's worth noting that modern meta-science had its blossoming in psychology in the 1960s, with the development of meta-analysis (with educational psychology and clinical psychology). Technically the origins are much earlier, in the 30s(?) in statistics, but as a field I think it took off around that time, and spread.

Similarly, the replication crisis was being discussed in a lot of areas, especially in psychology, throughout this time, but was largely ignored until after the Bem ESP study. Registered replications aren't new, nor is concern about meta-science; it's just had renewed focus in recent years for various reasons.

It's not all that surprising to me that meta-science is associated with psychology. After all, not only is psychology often sort of fuzzy (by necessity of its subject matter), but it's the science of human behavior, which I think can lay claim to scientist behavior as well.

I think it's arguably the greatest contribution of psychology to the sciences in general.

[logicchains]

>When you think about it, it's insane that metascience isn't studied more. We're going to question every little detail about the universe, but we're just going to take on faith that peer-review and publication in journals is an effective method for weeding out "bad" science?

In a sense we do have this: engineering and finance. Engineering turns good hard science into new tools, machines and weapons, and Finance turns good (predictive) soft science into new ways to make money.

[jhap]

> In a sense we do have this: engineering and finance. Engineering turns good hard science into new tools, machines and weapons, and Finance turns good (predictive) soft science into new ways to make money.

I think this is a common critique, but I also think it is missing the point. What if the question of interest isn't so easily verifiable like in Engineering? Do we just throw up our hands and give up on those questions? [The alternative to good social science is not no social science, it’s bad social science](https://statmodeling.stat.columbia.edu/2021/03/12/the-social...).

Finance is also a bit tautological in this regard. It seems that often prediction models are impossible to disprove (e.g., our arbitrage method doesn't work anymore, the market updated). Yes good for putting skin in the game, but doesn't seem like it does much to advance our long-term understanding of humans.

[logicchains]

>What if the question of interest isn't so easily verifiable like in Engineering? Do we just throw up our hands and give up on those questions? [The alternative to good social science is not no social science, it’s bad social science](https://statmodeling.stat.columbia.edu/2021/03/12/the-social...).

Some things may well be complex enough that it's simply impossible, with the amount of resources available to the average university, to conduct a thorough enough study on a representative enough sample that accounts for enough confounding factors to make a statistically sound prediction that generalises. If this were the case for a significant proportion of the subjects of study of a particular field, then it might well be better to "give up" and admit we don't and cannot know, otherwise we're essentially creating a factory for bad science (as the available resources relative to the scope of the problem aren't sufficient to create good science, and there's no negative feedback to stop the bad science).

[ImprobableTruth]

If there's "good social science" that can't be used to make predictions, what differentiates it from bad social science?

[akiselev]

Its utility [1]. The social sciences study a lot of things that people in group A intuitively understand that group B can be completely ignorant of - say, for example, how to navigate a complex social structure like office politics in a modern workplace. Making any sort of predictions about intangible outcomes where the Hawthorne effect is in full effect is pretty much impossible since group A will respond to the new knowledge gained by group B, in effect changing the system we're trying to predict. Individual's psychologies respond to the changing psychology of the group in nondeterministic ways (at least, relative to our ability to collect data on input variables and internal state).

We can bikeshed what makes something a "science" till the cows come home but the philosophy of science and epistemology were not settled with Bacon and Popper - the end goal has always understanding in the broadest sense. Those studies have value as long as they help someone make sense of and adapt to the social systems they're in. It does mean though that those studies should be approached with extreme caution (see the decades wasted on string theory) and anyone basing their research off past results needs to carefully validate their assumptions.

[1] I think in this case "predictive" as a scientific term of art is too restricting. Social sciences often deal with very personal interactions that appear nondeterministic at the scale of a society but are relatively predictable when applied to a stereotypical office or school setting.

[ImprobableTruth]

I don't understand what difference you're trying to make between utility and predictive power. If you can give information on what approach in general will be better to approach office politics that is just a prediction. It doesn't mean that these predictions have to be always right, but if they don't have predictive power and are no better than a coinflip, that "understanding" is just a post-rationalization that doesn't provide any utility at all.

At the very least, it seems to me like the person I originally responded to would also disagree with judging social sciences for its "utility" - the article they linked specifically contrasted it with the natural sciences that "solve problems".

[akiselev]

For something to be predictive in a scientific sense, it has to be repeatable. There is so much variety in individuals and their environments that most social sciences have little repeatability - they mostly study affluent western college students who have time to volunteer for college psychology studies. However, if you're mostly an affluent western college student, chances are that you can take some value out of the studies because they're selected for your environment rather than humanity as a whole (which is what they purport to do by claiming to study 'psychology' rather than western college students specifically).

Closest analogy off the top of my head is psychiatric drugs: their efficacy is generally bottom of the barrel except for some group with factor X (each drug has their own unique factor X). For the vast majority of these drugs, we have no method of screening for whether a person has factor X - we don't even know what it is most of the time - so doctors have to go through a process of trial and error with patients until they find the right drug or combination. Once they do, it's like a night and day difference for the patient, yet if we applied the same standard of evidence for psychiatric drugs that we do for blood pressure pills, we'd never make any progress. A lot of the drugs look like they don't work in phase III and we have no way to predict which drug which help which patient but the patients figure it out with their doctors because they have actionable data, even if it isn't predictive in general.

[hpoe]

I think this is an important point, ultimately good science will produce, verifiable, testable, actionable results. Until you have that no matter how much math you use, how many lab coats you've got, no matter how many journals you publish in you're just sitting there playing with strings.

[jrochkind1]

Telling when this has happened can be less trivial than you would assume though. People (scientists even) were sure that phrenology produced verifiable, testable, actionable results for a generation or two.

In the long run it usually comes out, but the run can be longer than you think, and you may not be where you think in it with regard to any particular current theory. I wonder what things we know all "know" are proven by science will be dismissed by later generations. (I personally guess a lot of genetics-related stuff will be).

(Note that something doesn't need to be verifiable, reliable, or true to be "actionable". You can act on anything...)

[loup-vaillant]

A good start would be something like Probability Theory: the Logic of Science By E. T. Jaynes: http://www.med.mcgill.ca/epidemiology/hanley/bios601/Gaussia...

Also, the difference between a bad method and a good method, is that the good method makes more accurate, better calibrated predictions (that is, using it makes us better gamblers).

[EGreg]

It is, but you arent aware because metameta science isn’t covered enough

[throwawaysea]

There was a past discussion about peer review on HN: https://news.ycombinator.com/item?id=20607259

Another related discussion was about the grievance studies scandal, which also touches on peer review and academic rigor in journals: https://news.ycombinator.com/item?id=18127811

[tgbugs]

It isn't so much that it is not applied as that the pressure to generate data inside academic science makes it extremely difficult for grad students and postdocs to allocate the time to doing science on their experimental processes. They already only barely have time to do experiments on the system they are actually trying to study!

Engineering organizations inside major corporations usually actively engage in process improvement because they are resourced to do so.

[aeturnum]

If you are interested in this I really recommend you look at writings from the fields that have descended from what was called "Laboratory Studies" and now variously calls itself "Science studies", or "science and technology studies" or "science, technology and society" (STS is a common acronym). For the stuff that's very lab-focused I'd say you could start with Bruno Latour and Steven Shapin, both classics of the older guard of the field.

[logicchop]

When someone hits you with a truck-load of mathematical jargon and convoluted experimental setups, most people just end up fatigued and find themselves nodding to conclusions out of fear of looking stupid. And the problem is only amplified by the recent "Science Rocks!" attitude making the rounds in pop culture.

[jxramos]

I was recently imagining an experiment classification tagging system: "trial", "reproduced", "peer reviewed", etc. I could imagine this set of information landing on some wikipedia page and the experiment in question would gain a bunch of these tag badges as understanding of the phenomenon matures.

[Bayart]

Epistemology is an interesting field, but it's just *not part of the curriculum for engineering and experimental sciences types.

I only had a good introduction to it when I took it as an optional course in a humanities college.

[WalterGR]

*just not part

[Bayart]

Indeed !

[javert]

The incentive system that academic scientists live under explains why they don't push for studying metascience.

If that's going to happen, it has to come from outside the government-science complex.

Good luck with that...

[spullara]

Dedicated folks that just try and reproduce important papers would be amazing and valuable. Only time I have seen it happen at scale was during the Cold Fusion days.

[throwawayboise]

That's because if cold fusion had been proven to work, there were fortunes to be made.

Most science has very little market value.

[rvba]

Current "science" is often not interested in replication or even testing if experiments can be replicated. The culture is tk be quoted and "science but boring" is not quoted. On top of that there is publish or perish.

Also dont want to point fingrers but some scientists come from places where cheating is the norm.

[libraryofbabel]

There is a ton of current and past scholarship on the sociology, history, and philosophy of science, and peer-review and publication is actually a pretty hot topic in those fields. Although yes, perhaps it would be nice if that work was better funded, or if practicing scientists paid more attention to it rather than just repeating old myths about how science works.

For example, here’s a scholarly article on the exact question you mention - how and where peer review came to be seen as a guarantor of scientific quality: https://www.journals.uchicago.edu/doi/abs/10.1086/700070 (tldr: it wasn’t the 17th century Royal Society; it’s much more recent.)

[aeturnum]

Another classic on this subject is Shapin's "Pump and Circumstance" - which is also freely available:

https://scholar.harvard.edu/files/shapin/files/shapin-pump_c...

[ryanmarsh]

Similarly, I heard someone the other day assert that no one had done an double blind placebo controlled trial of the effects of FDA regulation.

[hannob]

That is probably true, but do you have any proposal how such a trial should be done?

RCTs are good when they can be done and I'm all for doing more of them and too often there's no good excuse for not doing them. But at some level things just get impractical.