Hacker News new | ask | show | jobs
by blueskin_ 4367 days ago
There are enough malicious bots that do follow robots.txt to make it still an important option for most sites.