Y
Hacker News
new
|
ask
|
show
|
jobs
by
boxed
1457 days ago
I've put in a hard block for all crawlers on all pages. Works for my scenario I think. Hopefully they don't lie in their user agent. Then it's going to be really bad.