Y
Hacker News
new
|
ask
|
show
|
jobs
by
rsbadger
1452 days ago
Exactly - seems the safest way is to explicitly block known bots by user agent from even reaching pages you don't want indexed.