Hacker News new | ask | show | jobs
by etiennebausson 598 days ago
Then they are, by definition, bots scriping the site for informations, and should start by the robots.txt