[mmastrac]

I found that I was getting random bot attacks on progscrape.com with no identifiable bot signature (ie: a signature matching a valid Chrome Desktop client), but at a rate that was only possible via bot. I ended up having to add token buckets by IP/User Agent to help avoid this deluge of traffic.

Agents that trigger the first level of rate-limiting go through a "tarpit" that holds their connection for a bit before serving it which seems to keep most of the bad actors in check. It's impossible to block them via robots.txt, and I'm trying to avoid using too big of a hammer on my CloudFlare settings.

EDIT: checking the logs, it seems that the only bot getting tarpitted right now is OpenAI, and they _do_ have a GPTBot signature:

    2024-10-31T02:30:23.312139Z  WARN progscrape::web: User hit soft rate limit: ratelimit=soft ip="20.171.206.77" browser=Some("Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)") method=GET uri=/?search=science.org

[atif089]

Did you implement this in your web server or within your application? I'd love to see the code if you're willing to share

[mmastrac]

Yep, it's open-source:

https://github.com/progscrape/progscrape/blob/master/web/src...

Here's where we handle the rate limits:

https://github.com/progscrape/progscrape/blob/master/web/src...

I actually misremembered my implementation. It's rolling counting bloom filters, not a token bucket. :)