[leafo]

Same thing has happened to me: https://twitter.com/moonscript/status/1124888489298808834

The network address range falls under Facebook's ownership, so I don't think it's someone spoofing. I do think it's very possible someone found a way to trigger crawl requests in large quantity. Alternatively, I would not be surprised it's just a bug on facebook's end.

[_lqaf]

They've done this before to me, too. First I tried `iptables -j DROP`, which made the machine somewhat usable, but didn't help with the traffic. After trying a few things, I tried `-j TARPIT`, and that appeared to make them back off.

Of course, sample size of 1, etc. It could have been coincidental.

[hinkley]

Tarpits are an underappreciated solution to a pool of bad actors.

You can add artificial wait times to responses, or you can just route all of the 'bad' traffic to one machine, which becomes oversubscribed (be sure to segregate your stats!). All bad actors fighting over the same scraps creates proportional backpressure. Just adding 2 second delays to each request won't necessarily achieve that if multiple user agents are hitting you at once.

[apocalyptic0n3]

I never looked into the TARPIT option in iptables before reading your comment. That seems really useful. I've been dealing with on and off bursts of traffic from a single AWS region for the last month. They usually keep going for about 90 minutes every day, regardless of how many IPs I block, and consume every available resource with about 250 requests per second (not a big server and I'm still waiting for approval to just outright block the AWS region). I'm going to try a tarpit next time rather than a DROP and see if it makes a difference.

[rocho]

Be careful, as tarpitting connections can consume your resources faster than those of the attacker.

[hinkley]

Most spiders limit the number of requests per domain, so if it's stupidity and not malice, you probably don't have a runaway situation.

... unless you're hosting a lot of websites for people in a particular industry. In which case the bot will just start making requests to three other websites you also are responsible for.

Then if you use a tarpit machine instead of routing tricks, the resource pool is bounded by the capacity of that single machine. If you have 20 other machines that's just the Bad Bot Tax and you should pay it with a clean conscience and go solve problems your human customers actually care about.

[noir-york]

Only if you use conntracking though no?

[justinclift]

Interesting. Implemented DROP for some Cloudflare net blocks a few weeks ago due to lots of weird interrupted TCP connections from them.

That's seemed to be good enough, but will consider the TARPIT option if it turns out to be needed. :)

[jschwartzi]

one man’s weird interrupted TCP connection is another man’s nmap attempt.

[fomine3]

TIL TARIPT!

[zackb]

This was happening to us > 5 years ago. The FB crawlers were taking out our image serving system as we used the og:image thing. What we did was route FB crawler traffic to a separate Auto Scaling Group to keep our users happy while also getting the nice preview image on FB when our content was shared. I can't understate the volume of the FB requests, I can't remember the exact numbers now but it was insane.