[arohner]

You don't really have to get the diagnosis correct, you just have to get the error bars correct, and then disclose all of that.

"Watson believes the best treatment is Foo. It has an X% chance of working, with a confidence interval of Y. The second most likely diagnosis is Bar, with an expected loss of Z if the diagnosis is incorrect. Which would you like to do?"

Besides, several specialties are already essentially classifiers (radiologists, anesthesiologists). If machines can't beat humans at classifying cancer from lung scans, they're not far behind. By the time Watson is in production, I bet a computer will be replacing radiologists already.