|
|
|
|
|
by freddielarge
4368 days ago
|
|
fun fact: robots.txt can also be used by attackers to find admin interfaces or other sensitive tidbits that you don't want search engines to crawl lots of target-detection crawlers will look at robots.txt as the first thing they do to see if there's any fun pages you don't want the other crawlers to see |
|
That said, obscurity is not really security. Your admin pages should be behind a password, which, if coded properly, will exclude spiders, bots, and bad guys.