Hacker News new | ask | show | jobs
by hightrix 1050 days ago
> What if I want googlebot to crawl it but not bravebot?

Then you need to gate your content such that it is not available openly to the public.

This falls inline with many objections to Google's WEI. If you host content openly and allow access freely, then don't be surprised when people access it at will and use it for free.

1 comments

Then why does bravebot obey robots.txt at all? It does, and it will respect blocks of ggoglebot, but it won't allow blocking just it or just googlebot.