[snowcode]

another update: my suggestion of privately sending bots you want to "allow" to scrape your site, would make it trivial to ban bad bots. No guid, no enter.

Bots that flood your site, effectively DDOS the site, are no different to hackers trying to DDOS your site and would be dealt with as an illegal hacking attempt .. not a "legal" bot configuration issue.

It instantly puts all traffic into 2 buckets, legal and illegal. Versus 3 buckets of legal, badbot and illegal.

tools like "scrapeshield" et al already exist, so my suggestion would allow those vendors to make their scrapeshields even more focused, and allow me to build my poor man's version of those tools, since I dont want to pay for anything. even if it's $5 a month, or $5 a year per domain for some anti-scraping that's a deal breaker for me, since I'm managing hundreds of domains.