[meesterdude]

The thing about rate limits is, there are often hard limits. Try querying the site a bunch and see what the data transfer rate is. if it's consistently incapable of going above say, 128K/sec, you've got your rate limit.

Other times, they'll allow for bursts but sustained get rate limited - these can be detected just the same depending on their approach; if it's per connection (and not IP) you may have to find a big file to download.

It also helps to establish that traffic to other sites is not rate limited, and to use 3rd party checkers that check for connection rate across the globe. Or DIY and spin up a bunch of EC2 machines that are geographically diverse.

[rozab]

I don't know about this, I've heard that limiting is often intentionally inconsistent because this makes people less likely to visit. I think Harlow did some similar stuff with his monkeys.