[vegannet]

As a lay-person who reads scientific papers sometimes because they’re referenced on HN, how do I validate the credibility of a journal? Does a directory of journals exist with scoring, or is there a strategy I can use when evaluating a paper or journal to determine credibility?

[Blikkentrekker]

One doesn't validate the credibility of a journal; one validates the credibility of the methodology in the research.

And it then turns out that most methodology even in reputable journals are rather wanting with many objections that can be leveled against it.

A large amount of scientific research can't even be reproduced, and of much that can, even though the cold data can be reproduced, the conclusions that follow from the data are rather dubious leaps of faith.

It doesn't take much for something to be called “science”; it certainly doesn't take reproducibility, despite various claims to the contrary.

[PeterisP]

It absolutely makes sense to simply ignore articles based on the credibility of the journal, it's an effective and cheap first filter (and you absolutely need filters) - there are many predatory journals (like this one) which will publish anything that's paid for, they probably even outnumber "real" journals, and it makes all sense to automatically discard them without reading.

There is a lot of noise already in "proper" journals - but in the predatory journals, the signal-to-noise ratio is so extremely low, it's not worth looking into the credibility of the methodology of the paper because that's far more time and effort that the paper deserves. If it was any good, it would have been published in a better venue. If it could pass peer review, it would have been published in a venue that actually does peer review as opposed to these (many) predatory journals who just claim to do so. The authors have strong practical incentives to not publish it there if they can avoid to, and the fact that they chose to do so anyway indicates that no respectable place would publish it.

Because of that, if a paper is published in a place like this, is a completely reasonable prior to presume that overwhelmingly likely the paper is very bad, without even looking at the paper itself.

[Blikkentrekker]

This is true; it works in one direction, but not in the other.

But how the post which I replied to was worded suggested that one can automatically trust “science”, provided that it be published in a credible journal.

I also find that very often in the most reputable journals is where the most sensational papers end up before any attempt at reproduction has been made simply because the data they measured was far from the null hypothesis, which may entirely be a statistical fluke and not hold up under a reproduction attempt.

It is really quite easy to obtain spectacular data as a fluke.

[bonoboTP]

There are impact factors and top lists, and you can check out the Wikipedia page of the journal if it exists. You can google what are the "top journals for <field in question>". You can check the publisher. IEEE and Springer for example tend to be genuine.

But you still have to look at the individual paper and can't believe it just based on a single article. Research articles are for sharing results among experts (and for advancing the careers of researchers), they are not aimed at laypeople. It's easy to misinterpret them if you lack the background knowledge.

Even in non-predatory journals many results fail to replicate and are produced due to a publish or perish pressure. You're better off learning from textbooks so you get info that has been verified, digested and distilled and represent consensus. Cutting edge research proposes new ideas by one group of authors, it's not a consensus yet.

[Blikkentrekker]

> You're better off learning from textbooks so you get info that has been verified, digested and distilled and represent consensus.

A very large body of consensus ended up in textbooks that had either never been attempted te be reproduced, or was attempted, much later, and couldn't be reproduced, yet continued to remain in textbooks.

The truth of the matter is that most scientific research will never see an attempt at replication, though the replication crisis has no doubt influenced this culture, but before it, as little as 0.16% of peer reviewed results were attempts at replication, and most were unsuccessful. — of course, the successful ones were not as easily published, which is probably why no one attempted it.

[bonoboTP]

That may happen in exceptional cases, but less so for sciences with more concrete results. To learn about human psychology and society perhaps you're better off reading great novels than p-hacked sexy "research".

Regarding replication... Replication is boring in the eyes of funding committees. They want new and sexy results for their money, preferably at a steady rate of a bunch of papers each year.

[fabian2k]

As a very rough first filter, for papers in the area of natural sciences, check if the paper is indexed in Pubmed (https://pubmed.ncbi.nlm.nih.gov/). Every legimitate peer-reviewed journal in that area is indexed there, anything that is not indexed is almost certainly a dubious journal. But you can't use this the other way around, Pubmed still contains some journals that publish questionable stuff.

Beyond that you can look a bit at the impact factor of the journal, you can generally find it by just googling for the name of the journal + "impact factor" or on the Wikipedia page of the journal. But it is kinda hard to interpret this on its own, and it does not translate directly into credibility. A high impact factor shows that articles in that journal are cited very often, and you can assume that journals with high impact factors generally have a reasonable peer review system. But that doesn't say much for a single paper, reputable journals can easily fail in individual cases.

The Pubmed check generally filters out most of the journals that would just publish anything without real review, so it's the most useful filter in my opinion. I would not try to gauge more meaning than that from the journal and focus on the individual article.

[voqv]

Maybe it's an unpopular opinion, and biased by fields I've worked in as a grad, but I think you really just can't validate a particular paper without a background in the field. You need lots of context: recent works of that university department, credability/background of the last author post doc/professor, state of the art and cornerstone works in that field.

You could probably gain an entry level background (except for highly mathematical or medical things) by spending 5-10 hours a week for a few months reading various papers/online discussions. Assuming you have access to those forums. You should know thats the lower bound amount of literature reading full-time students have to do to keep up with their own field.

[tasogare]

> literature reading full-time students

Nobody keeps reading full time, that would not only be insane it would also be conterproductive since it’s time not spend actually doing something or writing papers. In most place it’s not necessary to know all the recent papers, and if a really important one is missing as a reference, a reviewer will note it in the comments.

[voqv]

I worded that poorly. I meant full-time students (as in they're not just reading about something out of curiosity while sitting on the toilet) that read literature as a part of their job.

[corty]

Actually, asking for the credibility of a journal is the wrong question. The real one is about whether an article is credible. But the answer to that is disappointingly complicated.

Articles are peer-reviewed for their readability (for peers, not laymen), consistency (e.g. if there is a mathematical proof in there, the reviewer might check it if possible. if there is an experimental technique, the reviewer might check if that technique could produce the results claimed) and reproducability (is the experiment described in sufficient detail? does the software provided produce the expected results? do the provided numerical results fit the claims?). All those checks are what the reviewer can do in a few hours, days at most. That is what peer-review means.

All the things to check beyond those are left as an exercise to other scientists after publication. Reproduction is a big task, usually almost as big as the original paper. Proofs are often extremely hard to check, so a few days by any old reviewer won't cut it. So if you want to know if the results in a paper are true, watch literature for the following years and look at publications citing that paper and (dis-)agreeing with its results. Same for (sometimes) letters to the editor, retractions and "everybody-knows"-rumours.

More reputable journals tend to attract better-quality papers (according to the peer-review criteria outlined above). That tends to correlate with a higher probability of the paper being true. But it is not a very strong correlation, there is utter nonsense, groupthink and polished turds even in very high profile journals.

[bonoboTP]

Right, I think many people believe peer review to be some big expert committee giving a paper lots of analysis and evaluation. In reality they are mostly PhD students with 1-5 years experience, not experts with decades of experience and one person may get like 5-10 papers to review at a time. Oh and it's unpaid work that gets time away from your own research and is not really incentivized beyond a vague sense of moral duty towards the progress of mankind's knowledge. The end result is that it's mostly pattern matching, does it look like the typical paper in the field? The gut reaction and impression strongly influences the decision, then the actual review is about justifying that decision.

I mean it's not totally arbitrary, really good reviewers do give it like 2-5 hours, but it's best thought of as a rudimentary filter rather than a meticulous verification.

[morsch]

There are a number of metrics[1] for journal ranking. The journal in question[2] claims to have (or have had) an impact factor[3] of 0.593 (2017-18); "the ratio between the number of citations received in that year [...] and the total number of "citable items" published in that journal[...]". That's not a very good score. Nature has an impact factor of ~42 (2019), or to pick another example, because I recently referred to it here on HN, Eurosurveillance has an impact factor of ~6.

I don't think blindly following these metrics is a good idea, but it's not as a bad first approximation.

[1] https://en.wikipedia.org/wiki/Journal_ranking

[2] https://juniperpublishers.com/ofoaj/

[3] https://en.wikipedia.org/wiki/Impact_factor

[jonnycomputer]

Impact factor works if you just to minimize the number of "bad" publications you look at. But if you rely on it too much, you are going to exclude lots of good publications too. Specialist journals tend to have lower impact factors, for one. Journal of Artificial Societies and Social Simulation is a good journal, but its impact factor has never been especially high.

[Phemist]

One possible way is looking at the list of editors of the Journal, and seeing if the researchers in question have back-mentioned that Journal on their CV (or personal homepage). The fact that researchers/peers in a field are willing to openly connect their prestige to those of a journal's, is usually an indicator that there is at least something to the journal.

Most if not all scammy journals will fail this test.. but it is also quite labor intensive.

[chabad360]

I believe you can probably get a good answer to your question with Beall's list: https://beallslist.net/

[FabHK]

^ That was the standard blacklist, but I thought it was shut down in 2017?

It's really tricky, because the move to open access publishing (which typically requires that authors pay) is a good thing on one hand, but facilitates these junk journals (and inflates publication count) on the other hand.

There are also Cabell's lists, both of predatory and "quality" journals.

https://en.wikipedia.org/wiki/Beall%27s_List

https://en.wikipedia.org/wiki/Cabells%27_Predatory_Reports

[nabla9]

When you see link to paper with interesting title posted to HN from nature.com domain, it's almost certainly from "Scientific Reports". It's not as bad as described here, but nobody publishes there if they have paper that is scientifically interesting. It's place to report what you did when nothing interesting came out of it.

Nature publishes many journals and "Nature" is the highest quality general science publication. "Scientific Reports" does just quick review for methodology.

[dash2]

A really simple good-enough approach is to go to Google scholar, type in the title, and count how many citations it gets. Adjust for years since publication. Many citations: probably important. Few citations: less important. No citations, more than a year old: probably not much good.

It isn't perfect. Different disciplines tend to cite at different rates; and you may believe that some disciplines are just cabals of bullshitters citing each other. I couldn't possibly comment.

[akvadrako]

You can start by making sure it has many citations, not self citations, and ideally also people challenging it.

Then you can judge the arguments of the paper and the papers referencing it.

Usually a paper with thousands of citations and only weak arguments against it is trustworthy.

[DataWorker]

Evaluate the authors directly.