|
|
|
|
|
|
by yakubin
1402 days ago
|
|
|
While robots.txt is not a good security measure, it's a good tool for preventing pollution of search results with pages like wp_login.php. More to steer users to the subset of pages that are useful to them, away from the pages which are only an implementation detail. Also, some poorly rate-limited crawlers actually abide by robots.txt, so it's useful to prevent unnecessary load. |
|
|