[withinboredom]

> if users click a lot, is it because they find many valuable things, or because they didn't find what they were looking for?

Why do you care as a search engine? This is a natural human problem that can't be solved with technology, only by humans.

It used to be, that I went to page 5 of Google instantly, because that was where the real results were. The first few pages were people who knew more SEO than sense.

These days, that doesn't work since "semantic search" because now it appears to be sorted by some relevance metric and by about page 5 you start getting into "marginally related to some definition of what you typed in but still knows too much SEO to be useful."

The point is, this was already a solved problem if you knew to go to about page 4-5. Then people started trying to use a technical solution to a very human problem.

[kqr]

> Why do you care as a search engine?

Wait, are you really asking why a search engine would care how well it finds what the user is looking for?

Granted, there are a lot of search engines that sell themselves on other metrics ("it's fast!" or "it uses AI!" or "it's in the cloud!") but any serious search engine player strives to learn how good it is -- in practise -- at helping the user find what they are looking for. That's ultimately the purpose of a search engine.

[withinboredom]

> Wait, are you really asking why a search engine would care how well it finds what the user is looking for?

While a useful metric, it's an unknowable metric.

1. You have no idea if the user even knows what they are looking for, so how would you know that they found it?

2. You have no idea if the user found what they are looking for, maybe what they are looking for isn't on the internet?

3. You have no idea if the user is even looking for something, maybe it was just a cat running across the keyboard?

The only way to learn the answer is to have humans talk to humans. You can't game your way through it by using metrics.

It reminds me of this one time the CEO asked our team to add a metric for "successful websites" (we were a hosting provider) and we rebuffed with "define successful." They immediately mentioned page views, which we replied "what about a restaurant with a downloadable menu that google links to directly?" and back and forth with "successful" never being defined for all verticals and all cases. It just isn't possible to define using heuristics.

[kqr]

I disagree. It's unfortunate that some users don't know what they want, some want things that don't exist, and that some are cats. But most users are humans with a rough idea of an existing thing they are looking for. It's worth it for a search solution to find out how good it is at helping them. The cats add noise to that measurement, they don't invalidate it.

Do you philosophically agree there are websites that are more successful than others? If yes, then there are tangible qualities that distinguish this group from the other. They may be subjective, fuzzy, and hard to pin down, but they're still there. If no, a success measure is irrelevant to you but other people might disagree, and once thoroughly investigated, you sort of have to agree the measurement coming out of it reflects their idea of success.

In none of this am I saying it's simple or easy (I started this subthread by saying it's difficult!) but fundamentally knowable.

Yes, humans talking to humans is definitely the start. But then I'm posivistically enough inclined that I think with effort we can extract theories from these human interactions.

[withinboredom]

I didn’t go into all the problems with “successful websites” but it really is impossible to measure. For me, my business site is successful when I capture leads, my blog is successful when I write posts, a restaurant is successful when people show up to eat. There’s no way of knowing what variables and metrics constitute success without asking the person.

I had a CEO who searched for the related business search terms every morning. No clicks, he just wanted to see the ranking. The other day, I was searching for an open NOC page that I knew existed but couldn’t remember the search terms. Eventually I gave up, but I’m 90% sure I left the tab open to a random promising search result that had nothing to do with what I was really searching for. There’s a pdf that archive.org fought over and simply mentioning it results in a DCMA, you can find it now, but for nearly 20 years, you could only find rumors of it on the internet and a paper copy was the only way you could read it.

Even when I know what I’m looking for exactly, I sometimes open a bunch of tabs to search results and check all of them, (This is actually the vast majority of my non-mobile searches) especially because the search results are often wrong or miss some important caveats — especially searching for error messages.

The only way you could find out these searches were unsuccessful (or successful) is to ask. There’s no magic metrics to track that will tell you whether or not my personal experience found the search successful.