Hacker News new | ask | show | jobs
by 1vuio0pswjnm7 410 days ago
CommonCrawl is not a vlaid comparison. Most robots.txt target CCBot.