[kews]

Do you know what proportion of the texty web instructs unknown crawlers to go away (or blocks them)?

[marginalia_nu]

It's hard to give numbers, it doesn't seem to be very many, but losing out on a few key sites does make a pretty big impact.

You see stuff like this sometimes, makes me a bit sad.

https://linux.die.net/robots.txt