Hacker News new | ask | show | jobs
by Quppa 3046 days ago
I don't mind Facebook crawling pages as long as it respects robots.txt, but for the last few weeks we've been hammered by requests from Facebook-owned IP addresses (millions of hits daily, 50+ for the same URL at times). They don't even set the User-Agent header.

There's a bug report regarding the missing header here: https://developers.facebook.com/bugs/1654459311255613/

Unfortunately it seems impossible to get in touch with Facebook devs directly.

2 comments

Send a cease and desist to the CTO. Wait 30 days, then sue under the CFAA. LinkedIn did it
No, LinkedIn was sued by the recipient of the letter.
what site do you own (if you can tell)?
Not sure I can say.

On a positive note, it's given us an opportunity to focus on performance improvements :)