|
|
|
|
|
|
by reconnecting
9 days ago
|
|
|
I'm not defending meta here, but I should mention that meta also uses crawlers to visit pages when someone send a link through their services. User-agent: *
can be ignored by bots, but if they ignore the disallow rule for their own UA, they can easily be blocked by network AS. |
|
|