[peterangular]

Yep I can come at someone with datacenter/residential/mobile/etc. IP addresses, all incredibly configurable to slip around network blocks. Luminati and Proxy Bonanza are the services I've used with the most success.

Getting source proxy lists of high-reputation networks is just $$ and a simple API integration game anymore.

[zenexer]

There’s also the issue of CGNAT. If you rate limit too strictly based on IP address, you harm users who are stuck with CGNAT, especially in Asia and Africa. India is particularly problematic.

As for stuff like Luminati, if you’re being sufficiently sneaky, chances are you’re not going to snowball in the first place. I’m not sure why anyone would bother paying for Luminati to crawl sites like the one for which I work, but I have seen people use it to scam.

We can’t really be bothered to waste resources blocking well-behaved crawlers. Just keep it at a reasonable pace, respect errors (especially 429, but also 410 and 503), and ensure we have a way to contact you if things go wrong.

[peterangular]

Yep - I have the HTTP error code detection dialed into an extreme because it's dumb to run a broken scrape anyway.

Frankly just any errors - if I see more than say 5-10 jobs fail within a 2-3 minute time period things are designed to wait X time, try again... and stop if they're still encountering errors and ping me to come in and investigate.

Faulty retry logic is just as dangerous as the forked/distributed run-off situation.