Y
Hacker News
new
|
ask
|
show
|
jobs
by
lakomen
908 days ago
Is there a "export all to robots.txt disallow" feature? If not, that would be great to have
1 comments
gavinhking
907 days ago
Yup, here is an example with all listed agents, although you probably want to modify it to not fully block all of them:
https://darkvisitors.com/robots-txt-builder
link