|
|
|
|
|
by fixermark
4093 days ago
|
|
This example actually highlights an interesting difference between the two. I think one of the reasons robots.txt is generally respected is that there's a stick behind that carrot; hypothetically (what with us all using so much cloud these days), a site administrator that noticed a traffic spike commensurate with something ignoring robots.txt can respond by treating the requests as attacker-originated, which most "legitimate" sites would want to avoid. What's the stick behind the carrot for do not track? |
|