[tastysandwich]

I can't say how the website I'm scraping would respond if I just went full throttle. But it's also just a matter of courtesy anyway not to make ten thousands requests per second.

Funny story - at work we once had a huge spike in requests from a single IP. We all crowded around, thinking it was some malicious hacker from France. How exciting - we're now interesting enough to warrant a DoS! Turns out another team in the company was just pulling all our data into Algolia to improve search. They were clearly not very courteous!

So on the other end (building APIs) I certainly do pay attention to traffic and have Grafana alerts set up around it.

[coldpie]

I was asking about randomizing the interval between requests.

[pablosr11]

tends to be around not repeating same patterns.

[cxr]

Boy, this subthread has been exasperating to read.