[Boywithhalo]

I agree that h-index isn't very indicative, but max citations seems like a poor alternative.

There is a relation between citations and certain aspects of research quality, but citation count can be easily gamed, and is affected by many factors other than the quality of the work itself.

Trying to use single measures to quantiy something as complex seems flawed to begin with.

[ItsMonkk]

Exactly.

> When a measure becomes a target, it ceases to be a good measure. - Goodhart's Law

> The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor. - Campbell's Law

You want to use metrics as a very broad breadth first search to help cull the search space and use trust systems as a depth first search. But once you have found signal through trust, you should completely ignore that first search and even look more into things that you had previously excluded.

Find researchers that you agree with, find the papers that they recommend, read those papers and check that you agree with them. If they recommend something from a no-name university, look at those first.

Unfortunately if this is not your field and you aren't able to determine quality, this becomes impossible. If it's important to you, you need to learn it. This is why I don't like non-technical managers. If the people who approve the grants do not understand the result, this is inevitable. It might work early on when trust still lingers, but as metrics take over the social systems always fall apart.

[mirker]

Top N papers is already a metric used for professorship slots, so Max or other functions are me simplifying the situation.

I agree though that citations are proxies for goodness and are ultimately poor indications of work. For example, a paper closing the field would have no follow-up work, but a provocative paper would have plenty (regardless of quality).

I think a quantitative metric is necessary because many decisions eventually hit a bean-counter (e.g., admin type), who is not able to assess research quality in the time they have. But even with experts reviewing a set of candidates on research qualifications, the panel would be unable to objectively provide a full ranking of candidates.

A perhaps better way to do it would be to perform proper accounting on research output which stabilizes across a 10-year timeline or so. Each researcher is a “stock” and the stock-price reflects current and projected “revenue”. If someone publishes only papers that go nowhere, their stock tanks. At top institutions, tenure kind of does this (only one time).