Hacker News new | ask | show | jobs
by geekishmatt 3765 days ago
robots.txt is the "(Web) Robots Exclusion List".

Any web robots should parse your /robots.txt before accessing any site, but it's still the robot's (or its programmer's) choice to use it.

So: No, but still add it to your site and maintain it.