[kqr]

Yup. This is the Long Click metric.

Evaluating search is difficult because it's a tension: if users click a lot, is it because they find many valuable things, or because they didn't find what they were looking for?

If a user clicked just once, is it because they found what they were looking for or just that the rest of the results were so bad the user gave up?

The long click (user clicked, then didn't click again for a while) is a better metric, but also not ideal: did they stay because they found what they were looking for, or was the result just that confusing they had to stay to comprehend whether it was the right thing? Most often it's because they found what they were looking for, but the pathological cases hide in the middle: many similar correct results, winner is the one that makes the user a little slower.

(This has nothing to do with tabs or back buttons, by the way. It happens any time they can detect subsequent clicks on the search result page.)

I've worked in the search space (though on less evil projects than Google) and I still struggle with the question on how to evaluate search. If you have ideas, let me know!

[plank]

One idea, but people will probably hate me for it: If you return to e.g. the google search site (hence: when the long click metric would be triggered) have a dialog on top saying ‘result great / OK / bad-or-confusing’. Can probably be gamed (bot nets trying to destroy the reputation of others) but at least a long time would not automatically mean ‘great result’. (In the arms race to combat destruction, it could be so that a ‘bad-or-confusing’ click would not actually push a value down, just not make it go higher).

Kind regards, Roel

[squidbeak]

This was tried with a +1 button around the time of Google Plus's launch.

[withinboredom]

> if users click a lot, is it because they find many valuable things, or because they didn't find what they were looking for?

Why do you care as a search engine? This is a natural human problem that can't be solved with technology, only by humans.

It used to be, that I went to page 5 of Google instantly, because that was where the real results were. The first few pages were people who knew more SEO than sense.

These days, that doesn't work since "semantic search" because now it appears to be sorted by some relevance metric and by about page 5 you start getting into "marginally related to some definition of what you typed in but still knows too much SEO to be useful."

The point is, this was already a solved problem if you knew to go to about page 4-5. Then people started trying to use a technical solution to a very human problem.

[kqr]

> Why do you care as a search engine?

Wait, are you really asking why a search engine would care how well it finds what the user is looking for?

Granted, there are a lot of search engines that sell themselves on other metrics ("it's fast!" or "it uses AI!" or "it's in the cloud!") but any serious search engine player strives to learn how good it is -- in practise -- at helping the user find what they are looking for. That's ultimately the purpose of a search engine.

[withinboredom]

> Wait, are you really asking why a search engine would care how well it finds what the user is looking for?

While a useful metric, it's an unknowable metric.

1. You have no idea if the user even knows what they are looking for, so how would you know that they found it?

2. You have no idea if the user found what they are looking for, maybe what they are looking for isn't on the internet?

3. You have no idea if the user is even looking for something, maybe it was just a cat running across the keyboard?

The only way to learn the answer is to have humans talk to humans. You can't game your way through it by using metrics.

It reminds me of this one time the CEO asked our team to add a metric for "successful websites" (we were a hosting provider) and we rebuffed with "define successful." They immediately mentioned page views, which we replied "what about a restaurant with a downloadable menu that google links to directly?" and back and forth with "successful" never being defined for all verticals and all cases. It just isn't possible to define using heuristics.

[kqr]

I disagree. It's unfortunate that some users don't know what they want, some want things that don't exist, and that some are cats. But most users are humans with a rough idea of an existing thing they are looking for. It's worth it for a search solution to find out how good it is at helping them. The cats add noise to that measurement, they don't invalidate it.

Do you philosophically agree there are websites that are more successful than others? If yes, then there are tangible qualities that distinguish this group from the other. They may be subjective, fuzzy, and hard to pin down, but they're still there. If no, a success measure is irrelevant to you but other people might disagree, and once thoroughly investigated, you sort of have to agree the measurement coming out of it reflects their idea of success.

In none of this am I saying it's simple or easy (I started this subthread by saying it's difficult!) but fundamentally knowable.

Yes, humans talking to humans is definitely the start. But then I'm posivistically enough inclined that I think with effort we can extract theories from these human interactions.

[withinboredom]

I didn’t go into all the problems with “successful websites” but it really is impossible to measure. For me, my business site is successful when I capture leads, my blog is successful when I write posts, a restaurant is successful when people show up to eat. There’s no way of knowing what variables and metrics constitute success without asking the person.

I had a CEO who searched for the related business search terms every morning. No clicks, he just wanted to see the ranking. The other day, I was searching for an open NOC page that I knew existed but couldn’t remember the search terms. Eventually I gave up, but I’m 90% sure I left the tab open to a random promising search result that had nothing to do with what I was really searching for. There’s a pdf that archive.org fought over and simply mentioning it results in a DCMA, you can find it now, but for nearly 20 years, you could only find rumors of it on the internet and a paper copy was the only way you could read it.

Even when I know what I’m looking for exactly, I sometimes open a bunch of tabs to search results and check all of them, (This is actually the vast majority of my non-mobile searches) especially because the search results are often wrong or miss some important caveats — especially searching for error messages.

The only way you could find out these searches were unsuccessful (or successful) is to ask. There’s no magic metrics to track that will tell you whether or not my personal experience found the search successful.

[chillfox]

I feel like the problem is trying to turn human experience into a metric. Probably the better approach would be to have a well staffed QA team.

[makeitdouble]

We should be mad at Yahoo for having fucked up. If anything, they could have spun out the search part and be remembered for it,.

[agileAlligator]

I honestly don't think it's possible to have a QA team large enough to handle the gajillions of websites that come up and disappear every day. They just have to come up with better and better metrics until they find one that approximates the human experience the best.

[sojournerc]

You don't have to cover the long tail... Maybe just top 10% of topics would be a big improvement.