Hacker News new | ask | show | jobs
by rsbadger 1452 days ago
Exactly - seems the safest way is to explicitly block known bots by user agent from even reaching pages you don't want indexed.