[embedding-shape]

Nah, just add a rate limiter (which any public website should have anyways). Alternatively, add some honeypot URLs to robots.txt, then setup fail2ban to ban any IP accessing those URLs and you'll get rid of 99% of the crawling in half a day.

[kstrauser]

I gave up after blocking 143,000 unique IPs hitting my personal Forgejo server one day. Rate limiting would have done literally nothing against the traffic patterns I saw.

[embedding-shape]

2 unique IPs or 200,000 shouldn't make a difference, ban the ones that make too many requests automatically and you basically don't have to do anything.

Are people not using fail2ban and similar at all anymore? Used to be standard practice until I guess before people started using PaaS instead and "running web applications" became a different role than "developing web applications".

[nijave]

It makes a difference if there's 143,000 unique IPs and 286,000 requests. I think that's what the parent post is saying (lots of requests but also not very many per IP since there's also lots of IPs)

Even harder with IPv6 considering things like privacy extensions where the IPs intentionally and automatically rotate

[kstrauser]

Yes, this is correct. I’d get at most 2 hits from an IP, spaced minutes apart.

I went as far as blocking every AS that fetched a tripwire URL, but ended up blocking a huge chunk of the Internet, to the point that I asked myself whether it’d be easier to allowlist IPs, which is a horrid way to run a website.

But I did block IPv6 addresses as /48 networks, figuring that was a reasonable prefixlen for an individual attacker.

[LtWorf]

If only it were that easy.

[macintux]

And for many people, "easy" is hardly the word to describe that.

No wonder small businesses just put their information on Facebook instead of trying to manage a website.

[LtWorf]

I mean, people are held hostage by "professionals" that will set up some overcomplicated backend or vercel stuff instead of a static single html page with opening hours and the menu.

[nijave]

The poison's also the cure! Just ask AI for a haproxy rate limit config

[LtWorf]

It will give one. Will it work? No. You seem to not understand that AI crawlers mask as multiple clients to avoid rate limiting and are quite skilled at that.