[terra_t]

92% accuracy, unfortunately, isn't good enough.

Bag-of-words models perform pretty well at classification and search, and the main thing you need to improve search is to boost scores when words are close together.

You might think you could improve performance by using semantically better defined features, but even 92% accuracy adds enough noise to foil your plans.

It's a big problem in A.I. systems that have multiple stages. You might have 5 steps in a chain which are each 90% accurate, but put them together and you've got a system that sucks. Ultimately there's a need for a holistic approach that can use higher-level information to fix mistakes and ambiguities at the lower levels.

[endtime]

92% in general would actually be really good for word sense disambiguation, but..."Apple" is a really easy choice. I'd like to see how he does with a trickier word like "right" (as in civil, vs. not wrong, vs. not left).

[terra_t]

Yes, it is good, but not good enough for many applications. You're also left with the issue that one kind of "apple" is more common than the other kind of "apple" so the baseline accuracy of something that always assumes it's one kind of apple might be surprisingly good.

That said, text-to-speech is a system where it's important to do disambiguation of a particular set of words. For instead,

"I read the news today, oh boy", "read" sounds like "red"

"I read the news every day", "read" sounds like "reed"

You need to be able to disambiguate the word sense to be able to correctly read the world "read". There are maybe 20 or so very common words that are like this, so a modest amount of work in this area would be part of a good TTS system.

[temugen]

Perhaps 92% is poor in a lot of scenarios, but for this type of approach it's a good accuracy (within some 1-delta confidence). The unfortunate part is that it really was just a simple naive bayes bag-of-words, and it's not surprising that it did so well on one test case (apple). Extending that to help general NLP in any way would be much more difficult.

[djacobs]

To be fair, he did this in a couple of hours.

[terra_t]

Sure, but the fact is that nobody actually needs a "word-sense disambiguator", they need a search system with better accuracy, or a classifier with better accuracy or an information extraction system that turns text into facts.

Many areas in NLP are like this. You can get 92% accuracy in a few hours of work, and then you can get 93% after a week or work, and then you can write a whole PhD thesis about how you got 94% accuracy.

To a certain extent, there are approaches, such as the Support Vector Machine that are "unreasonably effective" but once you get past that, you often have to confront issues that everybody wants to sweep under the rug to make a real breakthrough.

For instance, there was that NELL paper that came out a few months ago; NELL extracted facts from text but it had no idea that "Barack Obama is the President of the United States" was true in 2010, and that "Richard Nixon is the President of the United States" was true in 1972. If you can't handle the fact that different people believe different things and that statements have expiration dates, no wonder you can only get 70% accuracy in IX