Hacker News new | ask | show | jobs
by smilliken 1830 days ago
Googlebot also doesn't respect crawl delays in robots.txt.