I ended up quitting Kagi over that and built my own search engine for my own use. But if the comments here are correct that Kagi are trying to build their own index that replaces all the others, that would be great news. It seems lots of people still like Kagi regardless of that too.
For anyone trying to build their own, turns out SQLite can take you much further than it seems you should ever be able to. A metasearch layer fills in for everything else.
I am trying this. I noticed I only browse a subset of sites a lot. Also HN works wonders for finding links. But how do you get around all those bot measures/cloudflare nowadays? It seems only google IPs get a special pass.
Yep, discovering that you only need a subset of sites is the key. In my case, I still had my entire Firefox browsing history in its SQLite cache, so I tried to index every page I'd ever visited, and had a list of the domains I actually visit. It's the same insight Brave Search had - they used to be a metasearch engine, but they built their own index until they could serve 99% of queries directly, even though their index was vastly smaller than Google or Bing.
I struggle with indexing too. I don't do any crawling, just indexing from sitemap.xml files or URLs I add to the queue manually. Also I'm indexing from my own residential PC, not a hosted VPS. Something that helped me was Cloudflare's new crawler API. For all the sites where Cloudflare blocks you, just use their API instead:
Also for a quick index bootstrap, the Curlie database can be useful. It's the old DMOZ directory, the open Yahoo competitor, with 1.4 Million websites. A lot of the links are now very outdated, but it can be useful, and it's less than 500MB when converted to an SQLite database.
Unfortunately, that's nearly identical to the text from September 2024, with the only change being an increase in the number of external search sources:
I'm aware of their Teclis index, I used to be a paying API customer. Teclis is very small. It's primarily an index of indie blog websites (smallweb) mostly crawled via RSS feeds. That itself is a very cool idea, and a great supplemental index to have! But it isn't the kind of index that could ever replace their Brave / Yandex / SerpAPI dependencies. The Teclis API is even supplemented with results from Marginalia Search, a much larger index created by one person with less funding.
There's some technical info here on how Teclis is indexed, eg Readability.js for content extraction & Elasticsearch for the full-text indexing:
Kagi's short infoline about index distribution sounds new to me though. I would love to see a blog post from Kagi about that & where they're at with query coverage.
https://kagifeedback.org/d/4727-option-to-choose-or-exclude-...
I ended up quitting Kagi over that and built my own search engine for my own use. But if the comments here are correct that Kagi are trying to build their own index that replaces all the others, that would be great news. It seems lots of people still like Kagi regardless of that too.
For anyone trying to build their own, turns out SQLite can take you much further than it seems you should ever be able to. A metasearch layer fills in for everything else.