Hacker News new | ask | show | jobs
by johnny99 2471 days ago
Compliance with robots.txt is and always has been voluntary, and many crawlers have long ignored it, including Archive.org.