Hacker News new | ask | show | jobs
by brudgers 35 days ago
‘Keeping crawlers accountable’ sounds like whack-a-mole.

At least to me because benign crawling is likely to be a tiny subset of all crawling activity that is not done by google, etc.