[londons_explore]

Access to the click stream is the big bit.

Ie. Knowing which users clicked which search results.

Without the click stream, one cannot build or even maintain a good ranker. With a larger click stream from more users, one can make a better ranker, which in turn makes the service better so more users use it.

End result: monopoly.

The only solution is to force all players to share click stream data with all others.

[ChuckMcM]

Click stream is useful, without a doubt. It isn't essential. We had already started the process at Blekko of moving to alternate ways for ranking the index.

That said, if you run the frontend as proposed, you get to collect the clicks. That gives you the click stream you want. If the index returns you a serp with unwrapped links (which it should if it was unbundled from a given search front end) then you could develop analytics around what your particular customers "like" in their links and have a different ranking than perhaps some other front end. One thing that Blekko made really clear for me that the Google idea that there was always the "best" result for that query (aka the I'm Feeling Lucky link) there was often different shades of intent behind the query that aren't part of the query itself. Google felt they could get it in the first 10 links (back before the first 10 links were sponsored content :-)) and often on the page you could see the two or three inferred "intents" (shopping, information, entertainment were common).

[lelandbatey]

I don't think that's quite true, as competitors like Kagi have been able to compete well with effectively zero clickstream (by comparison). It'll help, but it's not the make-or-break that the index is.

[cocoafleck]

I think a click stream isn't necessary, but Kagi is not a good basis for the argument in my opinion.

Kagi is a primarily meta search engine. The click stream exists on their sources (Bing, Google, Yandex, Marginalia, not sure if they use Brave). They do have Teclis which is their own index that they use, and their systems for reordering the page of results such as downranking heavy ad pages, and based upon user preferences (which I love).

https://seirdy.one/posts/2021/03/10/search-engines-with-own-... is a source I would recommend checking out if you are curious.

[londons_explore]

Kagi sends searches to other providers (Bing?) and then simply re-ranks the results, so they're effectively inheriting the click stream data of those other providers.