[wzwy]

This post reminds me of this excerpt from the book How to Measure Anything by Douglas Hubbard:

> What many organizations do to assess risk is not very enlightening. The methods I propose for assessing risk would be familiar to an actuary, statistician, or financial analyst. But some of the most popular methods for measuring risk look nothing like what an actuary might be familiar with. Many organizations simply say a risk is “high,” “medium,” or “low.” Or perhaps they rate it on a scale of 1 to 5. When I find situations like this, I sometimes ask how much “medium” risk really is. Is a 5% chance of losing more than $5 million a low, medium, or high risk? Nobody knows. Is a medium-risk investment with a 15% return on investment better or worse than a high-risk investment with a 50% return? Again, nobody knows because the statements themselves are ambiguous.

[…]

It is true that many of the users of these methods will report that they feel much more confident in their decisions as a result. But, as we will see in Chapter 12, this feeling should not be confused with evidence of effectiveness. We will learn that studies have shown that it is quite possible to experience an increase in confidence about decisions and forecasts without actually improving things—or even by making them worse.

For now, just know that there is apparently a strong placebo effect in many decision analysis and risk analysis methods. Managers need to start to be able to tell the difference between feeling better about decisions and actually having better track records over time. There must be measured evidence that decisions and forecasts actually improved. Unfortunately, risk analysis or risk management—or decision analysis in general—rarely has a performance metric of its own. 3The good news is that some methods have been measured, and they show a real improvement.