Y
Hacker News
new
|
ask
|
show
|
jobs
by
sharlos201068
454 days ago
Crawlers aren't interested in fake pages that aren't linked to anywhere, they're crawling the same pages your users are viewing.
1 comments
danielheath
454 days ago
Adding a disallowed url to your robots.txt is a quick way to get a ton of crawlers to hit it, without linking to it from anywhere. Try it sometime.
link