[skybrian]

It seems like companies don't have to evaluate it your way? They could informally try it out on a few searches and say, "seems to work better than the other one." This is how Google became popular.

Or maybe do an A/B test. It's a little more formal but doesn't require any search-specific knowledge either.

[mlthoughts2018]

This approach typically fails quite bad in practice. The subjective impression of success depends on the people around at the time of the decision and the set of queries they chose to inspect. As corner cases pop up with significant cost in production (e.g. surfacing nudity in an image search for a query where it’s highly inappropriate) you become less and less capable of understanding why or hacking business logic in to an already brittle system.

The same problem appears for A/B testing different result orderings. It all hinges on the metrics you chose. If you only looked at e.g. top 5 click through rate, and then later a page redesign brings the top 10 results above the fold, and 6-10 are garbage, you’re suddenly screwed with no systematic way to adjust underlying parameters of the search model that control these things, or really even to get analytics data about it because you felt you could just outsource to a place like Algolia and not have in-house expertise, or that some engineers without statistics backgrounds could just hack it.

[cglace]

What do you do when you don’t have the budget for a team of in-house experts and you need something that is good enough and you need it now.

[mlthoughts2018]

Well, what you don’t do is rush to buy a wrong thing because you’re desperate and it has good marketing.