[nobodywasishere]

> didn't realize Kagi had no aspirations to build their own general purpose index

Kagi employee here. We're actively working on building our own indexes beyond the limited ones we have now, not just a general index but also purpose built indexes for things like programming, etc.

[rpdillon]

I did not intend to spread misinformation here, and would like to hear more about the general-purpose index Kagi is working on. I had based my comment on several Kagi pages, but mostly https://help.kagi.com/kagi/search-details/search-sources.htm..., which mentions Teclis as Kagi's own index, but https://teclis.com/ makes it pretty clear that it's a "small web"-focused tool:

> Teclis is an attempt to surface the less known web, the web of creativity and self expression, the more humane web.

> Teclis includes its own crawl as well as results from Kagi Small Web index and results with permission from Marginalia Search.

> Teclis works best with broad queries such as 'machine learning', 'vegan diet', 'religion' etc..

Is there another crawler doing the general-purpose stuff?

[nobodywasishere]

Not in production yet, so will have more to share in time.

[rpdillon]

Fair enough, thanks for replying!

[esperent]

How broad will they be? Do you aim to ever have large scale indexing of the web?

[Xunjin]

Hey do you guys have posts or sharing about it? It would be awesome to see what you are trying to accomplish, maybe it's time to post on HN ;)

[echelon]

Hurry. Google might give up the ghost on its search product and maintaining indices on anything not geared for LLMs.

I'm not sure antitrust will help you.

[tokai]

So you will stop buying Yandex data at some point?

[sph]

I care more from breadth of the dataset than politics in my search engine, thank you very much.

For everybody else there’d Google I guess.

[ljm]

What are the challenges of doing that when so much of the internet has turned itself into SEO slop to fit Google's algorithms?

I imagine there is still a whole load of stuff out there on the internet that Google would never surface because it doesn't have enough adsense or whatever. Are you finding that?

[account42]

There is a very easy 90% solution to that: massively downrank everything with ads with some exceptions added as needed.

[nobodywasishere]

That's what SlopStop is for :) developing methodologies that scale for detecting slop.

I mean it sounds like that already has a lot of overlap with our Small Web indexing efforts, so that part of our indexing efforts could be an extension of that. A lot if this is still in development though so I can't speak on specifics just yet.

[Dwedit]

How do you build a search index in the days of Anubis pages everywhere?

[account42]

Anubis is easy, just use a whitelisted user agent or a headless browser if some sites disable that - you need one to index web app abominations anyway. Cloudflare and Google reCaptcha are bigger problems.

[majorchord]

https://roundproxies.com/blog/how-to-bypass-anti-bots/

[xnx]

> We're actively working on building our own indexes

Lip service. You'll have some token index of Wikipedia or something so you can say your results are "a blend of our own index and other sources".

[freehorse]

Wikipedia is prob in "other sources", as they actually say they have a direct license for it.

https://blog.kagi.com/waiting-dawn-search#:~:text=Wikipedia,...

[account42]

Lol everyone has a direct license for Wikipedia.

[nobodywasishere]

It's funny you say that as we just switched the Wikipedia widget over to our own internal index. We don't intend to stop there either.