Y
Hacker News
new
|
ask
|
show
|
jobs
by
Thorrez
463 days ago
The problem is many crawlers pretend to be humans. So to ban the rest of the crawlers by default, you'll have to ban humans.