Hacker News new | ask | show | jobs
by lakomen 908 days ago
Is there a "export all to robots.txt disallow" feature? If not, that would be great to have
1 comments

Yup, here is an example with all listed agents, although you probably want to modify it to not fully block all of them: https://darkvisitors.com/robots-txt-builder