[PaulHoule]

Ranking 'quality' is tough because it has multiple dimensions and aggregating multiple kinds of utility is problematic, see

https://en.wikipedia.org/wiki/Arrow%27s_impossibility_theore...

developing a good ranking function for bag-of-words full text search is about carefully balancing the attraction of larger vs smaller documents in the collection. If you try to aggregate several bad search engines you get a bad search engine, if you try to add more factors you get a bad search engine, instead you have to walk the path of BM25 or one of the more modern 'information theoretic' ranking functions. (e.g. precious knowledge I got from years of reading conference proceedings and still having no idea how to make a good ranking function then finally reading the right review paper that pointed out the two discoveries the conference made in the first 10 years!)

[lifeisstillgood]

Are you saying you do know the way to create a good ranking function? Does Arrows theroum stop that?

[PaulHoule]

I know how to make some ranking functions that are good in some ways. They are not good in other ways so I don't break Arrow.

The shadow of Arrow however is that it is a lot of work to get a real improvement over a tfidf with random characteristics and you are limited by an asymptote which is less far away than you'd like: the rel teams at Google and Bing cannot beat the p=0.7@1 barrier for text search because 0.3 of the time they will get your intent wrong.

There are ranking functions in Lucene that can be tuned up as I described by making a test data set and using the methods developed at the TREC conference. That kind of tuning really works. At that conference you might see people fight over a point or two of AUC and I don't know if users can feel that but I am sure users can feel the 15 point and more difference we were seeing from tuning.

The odd thing is that hardly anybody does it and the knowledge seems pretty obscure. (e.g. I have talked to people at Lucene, OpenText and other full text search vendors and they are much more impressed with having 10,000 connectors than with the search results being good.)

[dang]

What was the review paper?

[PaulHoule]

actually it is this book

http://istl.org/06-summer/review1.html