|
|
|
|
|
by geekishmatt
3765 days ago
|
|
robots.txt is the "(Web) Robots Exclusion List". Any web robots should parse your /robots.txt before accessing any site, but it's still the robot's (or its programmer's) choice to use it. So: No, but still add it to your site and maintain it. |
|