[j-pb]

That's why it's important to make it a web of trust and not a plain "upvote"-"downvote" system.

E.g. publishing a paper that gets positive reviews gives you some form of trust token that you can spend on other papers in a positive review.

Ideally there is no single global metric for trust, but trust is computed relatively based on who you trust, so that everybody gets a personalized quality and relevance score for each paper.

In a way you could also personalise your definition of trust.

E.g. do you want to rank papers highly that have positive reviews by a handfull of highly trustwothy individuals, or do you trust papers that have many positive reviews.

[throwawaygh]

That's just another popularity contest with all of the exact same perverse incentives. How do you think people get on conference committees / journal boards?

We should stop worrying about popularity contests (citation counts, prestige-signalling conference committees/paper venues, etc.).

When we decide whether to retain an academic as a visiting scientist, consultant, etc., I am always the voice in the room encouraging folks to disregard everything except for one question: after having read 3 or 4 of this person's most relevant papers, do we think they can help us with the specific problems we face?

When I sit on grant review committees I ignore all the academic prestige shit and ask: is this a real 10-15-30 year problem for a field, does that field matter to the funding agency's objectives, and does the description of the plan of attack have a good chance of success? I don't need to evaluate popularity contests to answer those questions.

Popularity contests are low-effort short-cuts for the low-brow scientist/program manager. Best to ignore them if you have real problems and need real solutions.

[j-pb]

The point of the subjective and individualized scores is to avoid that, but fair enough, if we could get academia to switch to an open funding scheme without the current paper mill incentives more power to that.

I fear that the scientific community is not mature enough to do this properly and truthfully without some gamification though sadly, and funding commitees will want some kind of metric for "non-technical" people.

But maybe the only solution is to pass out funding based on random lottery tickets.

[throwawaygh]

> paper mill incentives

In the USA, publishing a large number of low-quality papers is at best unhelpful for getting grants. It's probably (very!) harmful.

> funding commitees will want some kind of metric for "non-technical" people.

The stakeholders within funding agencies are always technical. The external stakeholders -- congressional staff and corporate executives -- could not care less about citations/paper counts.

Based on your view of how these things work, I'm a bit curious about your background here. Have you sat on a funding committee for a public funding agency? Do you know the names of any program directors? Have you served on an academic grant review committee for your employer? If so, I'm really curious to know what field you're in, since things appear to work quite differently there.