Hacker News new | ask | show | jobs
by Matt_Cutts 4954 days ago
That's interesting.

Gabriel, does DuckDuckGo's crawler have a distinct user agent? Can you talk more about how DuckDuckGo observes/respects robots.txt?