[Jach]

That's not at all what Kasparov said:

    My concern about its utility, and I read they would like it to answer medical questions, is that
    Watson's performance reminded me of chess computers. They play fantastically well in maybe 90% of
    positions, but there is a selection of positions they do not understand at all. Worse, by definition
    they do not understand what they do not understand and so cannot avoid them. A strong human Jeopardy! player,
    or a human doctor, may get the answer wrong, but he is unlikely to make a huge blunder or category error--
    at least not without being aware of his own doubts. We are also good at judging our own level of certainty.
    A computer can simulate this by an artificial confidence measurement, but I would not like to be
    the patient who discovers the medical equivalent of answering "Toronto" in the "US Cities" category,
    as Watson did.
    
    I would not like to downplay the Watson team's achievement, because clearly they did something most
    did not yet believe possible. And IBM can be lauded for these experiments. I would only like to wait
    and see if there is anything for Watson beyond Jeopardy!.

If IBM wants to fix the "Toronto" problem, have at it. But those sorts of "embarrassing" errors could be quite costly in medical situations. During the show they showed Watson's progression from really stupid answers very frequently to less frequently, which makes me personally believe their fundamental process is flawed (not necessarily irreconcilable) and their current algorithms are just a bunch of hacks thrown together on top of Google rather than something more sophisticated like Wolfram Alpha.

[scott_s]

Watson was not confident in that answer - only 30% (http://asmarterplanet.com/blog/2011/02/watson-on-jeopardy-da...). Had that been a normal question, it wouldn't have buzzed in. It only answered because Final Jeopardy is the only time when not answering and answering incorrectly have the same penalty.

[michaelcampbell]

> but I would not like to be the patient who discovers the medical equivalent of answering "Toronto" in the "US Cities" category, as Watson did.

Surprise, that kind of mistake happens far too frequently in the medical field now.

Why is Kasparov commenting on something so far out of his recognized area of expertise relevant anyway? I don't go to Knuth for advice on chess, nor Hawking for snarky banter on economics, etc. (Although if I had access to either of those 2, I might try it.)

[Jach]

Would Watson make that mistake happen more or less often? (Bringing in Watson can lead to blindly trusting or blindly ignoring the "stupid computer" depending on the doctor; seems like a problem with doctors rather than a lack of tools?)

> Why is Kasparov commenting on something so far out of his recognized area of expertise relevant anyway?

Isn't the asking obvious? (I won't comment on the relevance; people do and read many irrelevant things every day.) People asked for his thoughts 'cause he got beat by IBM's Deep Blue and he's had a lot of experience with computers in their relationship with chess (specifically combining humans and computers to make really strong opponents). People also asked for Ken Jennings' thoughts and AI isn't his expertise. And people recently asked Hawking for his thoughts on aliens...

[lmkg]

Part of 'the "Toronto" problem' is simply the format of the challenge. In Jeopardy, Watson can give at most one response, in that particular situation exactly one. In a medical diagnosis situation, Watson's responses wouldn't be so constrained. He could give a list of 20 possibilities, with confidence margins for each one, and even as far as a list of possible additional tests designed to favor one possible diagnosis over another. This sort of information, utilized by a competent doctor, has a far, far smaller potential for disaster than the "what is a lobotomy?" scenarios that people are scared of.

[jhamburger]

My point is that the "Toronto" error is constantly cited as proof that Watson is fundamentally flawed, when it's actually a fairly reasonable bug if you understand the process it goes through to reach the answer- It's just seen as a stupid answer because it misses a key filter that humans would pick up.

In the medical case, it's actually better for the answer to be obviously, embarrassingly wrong than slightly wrong. Like the other commenter said, people aren't going to be getting amputations for headaches just because Watson says so. There's much more danger in something like prescribing medications with a fatal interaction, something that a hypothetical "Dr. Watson" would pick up.

[ppod]

there is a selection of positions they do not understand at all. Worse, by definition they do not understand what they do not understand and so cannot avoid them.

This is almost certainly true for humans too in terms of general problems rather than specifically chess. There are probably concepts which we have so little understanding and comprehension of that we can't even see our own ignorance. Rumsfeld's known unknowns.