Y
Hacker News
new
|
ask
|
show
|
jobs
by
blueskin_
4367 days ago
There are enough malicious bots that do follow robots.txt to make it still an important option for most sites.