[KMag]

I worked at Google on search indexing at the time Yahoo switched from their own search engine to using Bing. At the time, by most of Google's own search metrics, Yahoo had a product superior to Bing. If Bing had been spun off as a separate company, or otherwise hadn't had access to Microsoft's deep pockets and default IE search status, it's likely Yahoo would have fared better.

[toast0]

I was at Yahoo during that time, although not in web search. From what I could tell, company leadership was frustrated with lack of growth in search market share, and didn't want to invest in it anymore.

Yahoo was running user studies where they would put Google results and Yahoo results side by side but switch the branding; while Yahoo results were ranked better than Google for most of the tested queries, results with Google branding ranked better than with Yahoo branding, regardless of whose results they were.

The plan was to just use Google, but the DOJ (or FTC?) put out guidance that that would be anti-competitive, so Bing was it. This might have worked out anyway, but the expected cost savings from outsourcing search didn't actually happen that I saw, but I left in late 2011, and stopped following closely after that. Web search was also linked with search ads, which Bing did poorly at too.

[KMag]

Google also ran similar user studies, sometimes between Google and other search engines, and sometimes between production Google and a proposed change.

One tough thing is there isn't one search quality metric. It's important to have the search results page look good with its snippets, and another thing to have people actually look at the linked pages and compare the usefulness of the linked pages.

Common vs. uncommon searches are also important. It's not difficult to write a search engine that badly over-fits on the most common searches. However, for market share, it's important to do well enough on the common searches that users don't leave, and do well enough on tough long-tail searches that you pick up users that leave other search engines on tough queries. The idea is to be pretty good at the common searches, but the best at the kinds of searches that cause people to try other search engines. Naive frequency-weighted metrics will get this totally wrong.

It's also more important to get useful information in the first 2 or 3 links. If Google links to the second-best link at result #1 and puts the best link off the first results page, but Yahoo puts the best link down at #7 and second-best at #8, the user may lose interest before following a really good link.

I don't think Google took the union of front-page search results between two competitors and asked humans to hand-order the (up to 40) pages for how well they fit the query. But, that seems like a good way to test the actual usefulness of search results. You'd probably especially want to keep track of the percentage of the top 3 search results that were filled by top-5 (guessing at 5) useful links.

Anyway, inside Google it was well-known that Yahoo was the competitor to worry about in terms of search quality.

[saalweachter]

I would note it is possible (and even likely?) that each search engine performed better for its own traffic-weighted query stream.