[D-Machine]

> The first thing an academic does is check where a paper is published, before even reading it. It's a crutch

IMO, academics that do this are not very competent, because we have plenty of research suggesting that higher-profile journals are in fact less trustworthy in many ways, or that there is no correlation at all between reputation and quality (see my other post here in this thread).

Yes, some trash journals publish all trash, but, beyond that, competent researchers scan the abstract, look at sample sizes and basic stats, and if those check out, you skip to the methods and look for red flags there. Also, most early publications will be on an arXiv-like place anyway so you can't look to reputation yet.

Likewise, serious analytic reviews like meta-analyses don't factor in e.g. impact factor or paper citations, since that would be nonsense. They focus on methodology and stats.

I really think we ought to shame academics that are filtering papers based on journal alone, it is almost always the wrong way to make a quick judgement.

[blululu]

I have seen more than one PI at an R1 universities with multiple Nature publications use this heuristic. I would not call them incompetent.

[D-Machine]

Do you not notice the circularity of your reasoning here?

Also I didn't say incompetent, I said "not very". More competent researchers make journal rep only a very small factor, and it is not via the "high rep = more trustworthy" direction (which is the bad heuristic), it is "pay-to-publish journals = not trustworthy" (better heuristic).

Once you have ruled out a publication being in a trash journal, reputation is only a very minor factor in consideration, and methodological and substantive issues are what matter.

[jhbadger]

It all depends on whether the paper fits the journal. Minor journals serve a useful service as a repository for minor results. And minor results are still worth publishing because they might provide a detail or technique later needed for a major result. The thing to be wary of is when you see a stunning result that should really be in _Nature_ or _Science_ in some minor journal. Why isn't it? Was it submitted there first and rejected? It would be nice if the history of a manuscript (and its peer review) stayed with a manuscript so you could see if the authors really corrected problems brought up by peer review or were just spamming journals with a flawed manuscript until they found one that published it.

[D-Machine]

Agree with all this. Once you've filtered / made decisions of quality based on the more substantive criteria, journal reputation can provide useful additional information / context. The case you mentioned is a good example.

[blululu]

There is no reasoning. I gave you a statement of fact:

I have personally seen highly talented and successful researchers use the heuristic of journal quality when looking at the state of their field. These people are highly competent by any standard. If you want to play word games with negation you could say they are not not very competent.

[emil-lp]

> IMO, academics that do this are not very competent, because ...

Where's the cry-laugh emoji when I need it.

Of course academics check where stuff is published. Please...

There are still real journals put there, although you might not know which is which.

[D-Machine]

Ah, look, another smug sneer that ignores the evidence I presented, and makes another circular argument (i.e. that because academics look at rep, this is justified, even though I provided evidence disputing this).

I know what journals are better / not. But reputation only is helpful in letting you ignore trash journals, once you are out of trash land, rep is just not a very meaningful factor, and you have to focus on methodology and substance.

[emil-lp]

Where's the evidence you presented?

What are some higher-profile journals that are in fact less trustworthy in many ways?

[D-Machine]

I literally said it was posted in this thread, and a quick Ctrl+F of my username on this page would have found you it in a half second: https://news.ycombinator.com/item?id=47249236