| There's combining the two and then there is combining the two and really getting better results. Almost nobody in the commercial search field is doing quantitative evaluation despite https://trec.nist.gov/ About a decade ago I worked on a neural-net powered search engine for patents that used a neural network to compress bag of words features into a 50-dimensional vector (think LDA on steroids) and used a patented algorithm to https://patents.google.com/patent/CA2829569C combine that result with the keyword vector. It kicked ass. When we put up a demo we got a call the first day from the USPTO wanting to buy it. The thing is that algorithm searches over the feature vector and the residual of that feature vector in the bag-of-words space. There's no danger of infringing that with a BERT-like model because there isn't any such residual. We tuned up w/ Gov2 data from TREC and that was essential for getting our parameters right. If I were trying that now I would use logistic regression to make a probability estimator that "this document is relevant" that uses multiple scores as input; I'm not so sure how great results you will get but it is a rational basis to get started. TREC specifically does not reward training a probability estimator because they historically have been interested in "long tail" results and they don't think a kid sticking up his hand really high because he is really certain it adds value, but there are a lot of things with IR that are really hard (alerting) because we're mostly not using probability estimators and the ones we do make aren't that great (e.g. I never see p>0.7 for a conventional search engine tuned up that way.) |