[efitz]

The problem is, that as soon as a paper is published in a peer reviewed journal, it can be used to justify public policy positions. Then a large number of people will points to the study, or even worse, to press characterizations of what a person thinks the study says, to support their preference on the public policy position. Immediately, anyone who has an alternate position on the public policy will be accused of being a “science denier.”

Almost no one is interested in having an honest discussion about whether or not the original paper actually says what it’s characterized to have said, and whether it was a good study in the first place.

So nowadays, when public policy is concerned, largely I disregard any scientific study that is introduced to support any position on the policy, and just do my own cost – benefit trade-off to determine my policy position.

[Muller20]

This is a problem with the journalism and politics, it's not really about science. No scientist would trust a result that depends on a single small sample paper. Those are just stepping stones that may justify further research for more robust evidence. This fact is quite clear to scientist and it's why most would discourage the general public (including smart engineers) from reading academic articles.

But in general, I agree with you. It's ridiculous when someone pretends to shut down a complex issue by citing a random paper. However, an expert can still analyze the whole academic literature on a topic and determine what the scientific consensus is and how confident we are about it.

[mike_hearn]

> No scientist would trust a result that depends on a single small sample paper.

Unfortunately, they would. There are papers with thousands of citations that don't even have data samples, just models based on assumptions.

[FrustratedMonky]

Citations alone doesn't indicate how it is used.

Maybe a lot of scientist really liked the proposed 'model' and/or some discussion on the assumptions.

and cited it in their paper proposing some additions or follow on work.

That is also fine.

[mike_hearn]

When I checked how people were citing these useless papers, almost invariably it would be in a sentence like this:

"Computational modelling is a useful technique for predicting the course of epidemics [1][2][3][4][5]"

The cited papers wouldn't actually support the statement because they'd all be unvalidated models, but citing documents that don't support the claim is super common and doesn't seem to bother anyone :( Having demonstrated a "consensus" that publishing unvalidated simulations is "useful", they would then go ahead and do another one, which would then be cited in the same way ad infinitum.

[moi2388]

I disagree. A scientist could read a single paper and find out n is small, or identify a flaw.

But there are loads of papers like this. Then you have some literature studies which look at all these papers together and get result aggregates.

Then you get some “proper” studies which link to these aggregates, and several small studies, and you’re going to read these “proper” studies which are quoted often and deemed decent or good quality.

And at no point will you realise it’s all based on shoddy foundations.

This is for example what recently happened in social psychology

[adolph]

> The problem is, that as soon as a paper is published in a peer reviewed journal, it can be used to justify public policy positions.

That’s not a science problem. That’s a political problem. Making choices based on a single paper that may not be replicated is the problem.

[cynicalpeace]

Same here, and the more "the science" is used as a cudgel, the more people will resent it, whether or not they know a lot of it isn't even replicable.

[davisoneee]

You say you do a cost:benefit...but where do those costs come from? To me, that's just voluntarily doing your own ignorant, sloppy science I was mentioning above. If you only consider the blatantly obvious costs and benefits, you are completely ignorant of any 2nd or 3rd order...or even your own blind spots. You may radically under or overstate, or even calculate in the wrong direction.

I think a better position is that we should have a higher bar of what level of study or replication is required for a given situation...whether that be health, housing, infrastructure or whatever policy is coming in...what kind of monetary outlay and timeline of impact is expected. I don't think most people here would be happy with a 6-person study, unreplicated, deciding policy...so what IS the threshold?

[efitz]

I don’t analyze the papers, I analyze the policies.

Take the climate. Assuming the science is correct (storms worse, oceans rising, etc), let’s do a cost benefit analysis of the proposed policies.

The proposed policies in the US all dramatically increase the cost of energy (and therefore of everything), but only slightly slow the progression of warming, AND the bigger contributing countries in the developing world, esp China and India, will continue or increase their CO2 output.

We already know how to build dikes, and people who buy oceanfront property already know the risks, and over the last century we as a species have gotten really good as reducing deaths due to weather, so I don’t support the draconian carbon reduction proposals, and instead we can just deal with the side effects as they arise.

[isotypic]

But how do you analyze the policies without doing science? Nothing in the above is sound to me.

"The proposed policies in the US all dramatically increase the cost of energy" - why? How do you even begin to conclude this without looking at some sort of (economic/scientific) analysis?

"only slightly slow the progression of warming" - again, how are you concluding this?

"we as a species have gotten really good as reducing deaths" - why should this trend continue? Why should it continue in the face of more extreme weather/climate change?

All I see are things you _think_ are true, and so to you your argument seems sound. But as the comment you replied to said, all I see is ignorant, sloppy science, since any meaningful analysis of these policies is by definition science. These cost/benefits you mention are not universal apparent truths.

[efitz]

I’m don’t need to analyze any scientific paper to form an opinion on a matter of policy; what scientific paper should I read to decide whether to support an increase on property tax in my county to support schools? The idea that I have to defer my judgment to someone in a lab coat is insulting.

As to your other questions, the answer to all (which any trivial web search would answer as I did) is “I can read”.

And I didn’t “cherry pick” some particular study or some blog that supports my preconceptions. All of the links I have provided are from sources that assume severe climate change is coming without serious interventions in greenhouse gas emissions; they are all on the “climate activist” side.

Dramatically increase the cost of energy, note that this is BY DESIGN, it’s the entire point of carbon taxes and markets: https://www.npr.org/2021/12/16/1064951646/why-the-cost-of-ca...

Only slightly slow the progression of warming: https://www.science.org/content/article/paris-climate-pact-5...

Weather related deaths down 98% in past 100 years: https://reason.org/policy-study/decline-deaths-extreme-weath...

And of course now I’m being down voted for my example.

I never challenged a single point of climate orthodoxy; I never questioned whether global warming was occurring, whether it was man-made, whether the studies and models predict warming accurately, or whether the impact will be as stated. I stipulated every single one of those points. But simply my personal conclusion that I don’t believe the policy trade-off is worth it, means that my voice is not worth hearing.

[cynicalpeace]

It's not the current threshold, where you just need a single paper in a semi-prestigious journal to become "the science". This was my point.

[davisoneee]

From my perspective, I don't think I ever see "the science" being decided based on a single crap paper, so it seems a bit of a skeptical straw man. Perhaps that's different in your country, state, city, county, continent, island. Perhaps that's true for a particular policy _area_ that you are invested in. In general, I find arguments on scarequotes "science" often overblown and just reinforcing existing biases.

How much 'policy' are you happy with that you've never checked to validate that it was properly replicated beforehand? How much of your skepticism is against "the science" on new policy you start with an initial dislike to...compared to policy that we consider as standard, are probably comfortable with, likely consider to be 'beneficial enough'...and yet haven't adequately scrutinized?

[cynicalpeace]

The COVID era has given us many rich examples of lies dressed up in a single paper.

https://www.science.org/content/article/many-scientists-citi...