[js4ever]

I have solved this kind of attacks with redis + app modification to count requests per ip per minute and auto add iptables rules to ban the offenders ip and deban it after xx minutes. Iptables rules are then synchronized to my fleet of front end servers.

I noticed Cloudflare is doing the same but 1 level deeper with XDP drop: https://blog.cloudflare.com/how-to-drop-10-million-packets/

[zenexer]

This person used a new IP address for every single request, so that won’t work. And that’s a growing trend.

[peterangular]

Yep I can come at someone with datacenter/residential/mobile/etc. IP addresses, all incredibly configurable to slip around network blocks. Luminati and Proxy Bonanza are the services I've used with the most success.

Getting source proxy lists of high-reputation networks is just $$ and a simple API integration game anymore.

[zenexer]

There’s also the issue of CGNAT. If you rate limit too strictly based on IP address, you harm users who are stuck with CGNAT, especially in Asia and Africa. India is particularly problematic.

As for stuff like Luminati, if you’re being sufficiently sneaky, chances are you’re not going to snowball in the first place. I’m not sure why anyone would bother paying for Luminati to crawl sites like the one for which I work, but I have seen people use it to scam.

We can’t really be bothered to waste resources blocking well-behaved crawlers. Just keep it at a reasonable pace, respect errors (especially 429, but also 410 and 503), and ensure we have a way to contact you if things go wrong.

[peterangular]

Yep - I have the HTTP error code detection dialed into an extreme because it's dumb to run a broken scrape anyway.

Frankly just any errors - if I see more than say 5-10 jobs fail within a 2-3 minute time period things are designed to wait X time, try again... and stop if they're still encountering errors and ping me to come in and investigate.

Faulty retry logic is just as dangerous as the forked/distributed run-off situation.

[CodesInChaos]

I'd assume they use 1000 distinct IPs because the system scaled to 1000 instances and presumably many of them came from related IPs. So it makes IP banning considerably harder, but not impossible.

[zenexer]

More importantly, any reasonable IP-based approach would have a lot of false positives. What if there’s an RSS service like Feedly running on the same cloud?

[CodesInChaos]

Depending on your product, blacklisting all AWS IPs might be acceptable. For example my company has a VPN exit on AWS, which appears to be blacklisted by twitter.