[mlthoughts2018]

While it’s good information retrieval work, I’m super skeptical that it actually adds meaningful value for customers.

Having worked on several different search engines at several companies, my consensus on this stuff is that customers usually just want more simple boolean filters or multi-choice filters that they can use to precisely control thevresult set for their preferences, and they want you to clean up the data backing the filters so they are highly accurate.

Spending huge money on a fancy system to try to understand their intent is overwrought. Did you ask them? Did you try just empowering them to communicate their own intent with more filtering or ranking options they can tune or manipulate on their own?

This stuff annoys me sometimes because it’s dripping with so much marketing hype. Look at a sexy new embedding-based recommender, or a reinforcement learning agent that learns how to rank to gamify placed orders.

Maybe you could have better served your users by not spending big salaries + compute costs on this crazy query understanding model and instead just cleaned up your data so that you stop getting the opening hours of restaurants wrong, or you stop giving back BBQ places when somebody checks “vegetarian.”

In so many cases, just some simple filters + data cleaning is worth way more to the customer than all this.

[amelius]

> Having worked on several different search engines at several companies, my consensus on this stuff is that customers usually just want more simple boolean filters or multi-choice filters that they can use to precisely control thevresult set for their preferences, and they want you to clean up the data backing the filters so they are highly accurate.

Were these customers very computer-literate?

[mlthoughts2018]

It varied depending on the domain. In one company it was a high traffic online merchant, and customers were basically a random sample of the middle-class population at large, no better-than-average-person computer literacy.

In another case it was a multimedia company and the search users were producers, artists and technical directors, with much more of a power user mentality to searching for what they wanted, and jobs that required basic computer literacy and maybe familiarity with special GUI programs.

But I’m telling you, in every focus group or A/B test or user session we ever looked at, at any of these companies regardless of the user characteristics, the absolute number one thing every time was user requests for more and more categorical filters to allow them to efficiently exclude huge classes of results they know are not relevant to them.

Amazon (I did not work there) actually gets this right, with lots of brand, size, color, Prime-eligible, etc. filters for everything. That stuff requires no machine learning, is simple to implement and A/B test and measure usage efficacy for, and people really want it.