|
|
|
|
|
by Quppa
3046 days ago
|
|
I don't mind Facebook crawling pages as long as it respects robots.txt, but for the last few weeks we've been hammered by requests from Facebook-owned IP addresses (millions of hits daily, 50+ for the same URL at times). They don't even set the User-Agent header. There's a bug report regarding the missing header here: https://developers.facebook.com/bugs/1654459311255613/ Unfortunately it seems impossible to get in touch with Facebook devs directly. |
|