Hacker News new | ask | show | jobs
by stanleydrew 5609 days ago
I'm pretty sure the Bing Crawler does respect robots.txt. The data Bing collected didn't come from spidering Google.