Hacker News new | ask | show | jobs
by sharlos201068 454 days ago
Crawlers aren't interested in fake pages that aren't linked to anywhere, they're crawling the same pages your users are viewing.
1 comments

Adding a disallowed url to your robots.txt is a quick way to get a ton of crawlers to hit it, without linking to it from anywhere. Try it sometime.