[baryphonic]

Implicit in all of this is the is-ought problem.[0] The data are collected and interpreted under some procedure, often with normative biases built in about how the world ought to be (especially when involving human subjects), but are interpreted as saying what the world is. Thus data collection is fertile ground for charlatans.

When the psychiatric profession or Google or whoever else use experimentation to decide on what criteria they should follow, with sound controls, valid statistical analysis and loads of replication, they either arrive at evaluation procedures without much bias or, more likely, they realize the phenomenon they're trying to measure is almost all noise with no or excessively weak signals.

A better approach would be to acknowledge as much normative bias as possible up front, then conduct tests using sound experimental design. But the problem with this approach is that the data shows performing a bunch of well-crafted experiments is expensive, and management doesn't buy in if the vast majority are unlikely to reject the null. That leaves us which a class of "data driven" managers who are in fact indulging their biases to a sometimes extreme degree, using "the data" as a shield.

[0]https://plato.stanford.edu/entries/hume-moral/#io