Y
Hacker News
new
|
ask
|
show
|
jobs
by
danielheath
454 days ago
Adding a disallowed url to your robots.txt is a quick way to get a ton of crawlers to hit it, without linking to it from anywhere. Try it sometime.