Hacker News new | ask | show | jobs
by osdotsystem 512 days ago
Me i dropped a bunch of dont allows for different bots, as many as i could find.
1 comments

I'm curious — if that's your attitude, why not all bots? You can say "User-Agent: foo\nDisallow:\n\nUser-Agent: *\nDisallow: /" to allow only named bots and reject them by default.