Y
Hacker News
new
|
ask
|
show
|
jobs
by
az09mugen
396 days ago
Totally agree, that's conceptually the same problem as robots.txt. As stated in
https://www.robotstxt.org/faq/blockjustbad.html
:
> But almost all bad robots ignore /robots.txt, making that pointless.