Hacker News new | ask | show | jobs
by danielheath 454 days ago
Adding a disallowed url to your robots.txt is a quick way to get a ton of crawlers to hit it, without linking to it from anywhere. Try it sometime.