Hacker News new | ask | show | jobs
by Capricorn2481 529 days ago
I don't understand this. You don't have routes your users might need in robots.txt? This article is about bots accessing resources that other might use.
1 comments

It seems better to put fake honeypot urls in robots.txt, and block any up that accesses those.
Blocking will never work.

You need to impose cost. Set up QoS buckets, slow suspect connections down dramatically (almost to the point of timeout).

Ah I see