[jonathaneunice]

This is an ancient technique. I used essentially the same process ~30 years ago (1987–95) to publish N-way product comparisons. The approach was old even then (see e.g. https://en.wikipedia.org/wiki/Quality_function_deployment).

The promise of this quantifying is that results are more objective and precise, so everyone can come to rational agreement. In practice, it doesn't do that. What attributes are chosen to quantify, how they're scored, how they're weighted—these are all subject to a great deal of fiddling and constant debate. "You over-weighted X!" or "You didn't consider Y!" Et cetera. Truly never-ending, and unless participants already well-aligned, doesn't secure genuine consensus.

Results are also highly perturbable. Tweak the weights and/or scores but a little and they tell an entirely different story. New winners emerge, clear victories become dead heats, and the former Red Lantern Award winner is suddenly in the middle of the pack.

[elefantastisch]

I've seen the same problems when this technique is attempted.

Something else to consider is that weights scaling linearly may not make sense.

More importantly, criteria may overlap. In the example in the article, I suspect technical ease and scaling ease are highly correlated, which effectively means you're double counting.

Basically the technique only works when you have fully independent criteria which cover the full spectrum of what matters and which can be weighted objectively using a scale that represents true relative importance.

[motohagiography]

When I have used this technique, I kept the weights confidential until I had collected the data from teams about their rating of each criteria on the strategies.

Then, I created two conversations. One was about our confidence on those estimated ratings together, and the second was the proposed weights and whose they were, which I revealed after so we could add up what the model provided and have a much faster discussion about whether it yielded what we ought to do.

Nobody wants to be subject to process, but almost everyone wants to appeal to it to get their way, so I wouldn't use this tool to make decisions themselves, but instead to make higher quality ones on teams faster.

[jackallis]

what's a better technique that has worked for you?

[jonathaneunice]

Discussion plus qualitative not quantitative scores, in simple terms like "Weak" and "Strong" (plus occasional modifiers like "Very"), with no weighting of attributes. Keep the comparison matrix simple and the discussion at a high level of granularity. While that's paradoxical for the analytic person I am, I've come to realize that "net scores" are usually not the point. The real goal and need is to have a multi-party discussion that leads towards consensus, or at least a rational understanding of why product/approach X was chosen over its competitors. In my experience, soft, non-numerical discussions of the choices available have more often lead to that happy (quasi-)consensus than when the numbers come out.