| Exactly. > When a measure becomes a target, it ceases to be a good measure. - Goodhart's Law > The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor. - Campbell's Law You want to use metrics as a very broad breadth first search to help cull the search space and use trust systems as a depth first search. But once you have found signal through trust, you should completely ignore that first search and even look more into things that you had previously excluded. Find researchers that you agree with, find the papers that they recommend, read those papers and check that you agree with them. If they recommend something from a no-name university, look at those first. Unfortunately if this is not your field and you aren't able to determine quality, this becomes impossible. If it's important to you, you need to learn it. This is why I don't like non-technical managers. If the people who approve the grants do not understand the result, this is inevitable. It might work early on when trust still lingers, but as metrics take over the social systems always fall apart. |