I am trying this. I noticed I only browse a subset of sites a lot. Also HN works wonders for finding links. But how do you get around all those bot measures/cloudflare nowadays? It seems only google IPs get a special pass.
Yep, discovering that you only need a subset of sites is the key. In my case, I still had my entire Firefox browsing history in its SQLite cache, so I tried to index every page I'd ever visited, and had a list of the domains I actually visit. It's the same insight Brave Search had - they used to be a metasearch engine, but they built their own index until they could serve 99% of queries directly, even though their index was vastly smaller than Google or Bing.
I struggle with indexing too. I don't do any crawling, just indexing from sitemap.xml files or URLs I add to the queue manually. Also I'm indexing from my own residential PC, not a hosted VPS. Something that helped me was Cloudflare's new crawler API. For all the sites where Cloudflare blocks you, just use their API instead:
Also for a quick index bootstrap, the Curlie database can be useful. It's the old DMOZ directory, the open Yahoo competitor, with 1.4 Million websites. A lot of the links are now very outdated, but it can be useful, and it's less than 500MB when converted to an SQLite database.
I struggle with indexing too. I don't do any crawling, just indexing from sitemap.xml files or URLs I add to the queue manually. Also I'm indexing from my own residential PC, not a hosted VPS. Something that helped me was Cloudflare's new crawler API. For all the sites where Cloudflare blocks you, just use their API instead:
https://developers.cloudflare.com/browser-run/quick-actions/...
https://developers.cloudflare.com/browser-run/quick-actions/...
Also for a quick index bootstrap, the Curlie database can be useful. It's the old DMOZ directory, the open Yahoo competitor, with 1.4 Million websites. A lot of the links are now very outdated, but it can be useful, and it's less than 500MB when converted to an SQLite database.