Hacker News new | ask | show | jobs
by lern_too_spel 1050 days ago
Then why does bravebot obey robots.txt at all? It does, and it will respect blocks of ggoglebot, but it won't allow blocking just it or just googlebot.