|
|
|
|
|
by Symbiote
6 days ago
|
|
I have observed the same from Meta's crawler. User-agent: *
Disallow: /
on e.g. our preproduction site, Meta is the only big-tech crawler that accesses it, at least with an honest user agent. (Meta also accesses disallowed paths on the production site.) |
|