Hacker News new | ask | show | jobs
by whazor 405 days ago
Deny-listing/banning bad crawlers is fine. Especially if they ignore the robots.txt.

But allow-listing particular crawlers only is collusion.