Y
Hacker News
new
|
ask
|
show
|
jobs
by
lern_too_spel
1050 days ago
Then why does bravebot obey robots.txt at all? It does, and it will respect blocks of ggoglebot, but it won't allow blocking just it or just googlebot.