Hacker News new | ask | show | jobs
by marginalia_nu 1542 days ago
> May I ask how you decide to add me content? Do you follow links? Do you use other search engines' results as a starting point?

I initially did basically a DFS-walk originating at a few websites I liked, with some filtering criteria that deprioritized websites that didn't look too interesting. Now that I have a fairly comprehensive mapping of the space I want to index, I use a few factors like frequent outbound links from highly ranking domains to inform which new sites to index.

> I could probably shoot many more questions, but don't want to be a nuisance.

No worry, I love to talk about this stuff.