Idk, if bots ate hammering your server then setup rate limits. If you have content that you don't want others to have access to, don't serve it with a webserver.
I used to just start giving any IP downloading way too much a redirect to multi-tb NASA images. This was a long time ago but it was surprisingly how many would follow redirects and never time out. Wouldn't see a request again for hours and then its right back to downloading a new part of the sky.
Those images also used to crash all the early GUI irc and chat clients that showed inline images without size checks...
How were you tracking each IP address's data usage? Did you parse the logs every request? Store usage in a database? At the application or webserver level?
Webalayzer! I'm not sure there were really any other options at the time other than writing your own. Parsed the apache logs and gave you pretty detailed results and you could see the usage (in kb, which tells you how long ago this was!) broken down by date and IP.
Once you added a redirect rule for the IP to apache you'd just check your log and see the IP that was hitting you every couple of minutes poofed for a good few hours.
This. What even is the point of blocking scapers if Google consumes your content anyway and serves it as an AI answer?
These are sad times we're living as far as openness of the web goes. People would have less of a scraping problem if their websites didn't ship with 20MB of JS.
There is something to be said for "one way indexes."
Imagine you run a company register for a local government. You want to let people look up companies by their registration number (which they must disclose in all communications to you) to see if they're legit and whether any warnings have been raised against them. You don't want unscrupulous marketers to just be able to `SELECT * FROM companies WHERE type='nail_salon' AND city='london'`.
If you aren't super strict about scraping, some shadowy business in Neverland, completely unconcerned with following your laws, will build that database.
> Imagine you run a company register for a local government.
Is this data not public for some reason? I think it will not hurt if there are multiple copies spread between public offices and private companies. What really hurts is a private company hammering your webserver for their own profit. They should get their own copy.
If the purpose of the index is to allow people to lookup registration and warnings, probably just serve the list. This is public information and doesn't need to be gated. CSV header could be:
I have blocked several Asian countries because their IP ranges kept sending stupid scrapers that repeatedly downloaded the same image with a made-up query, bursting through the basic cache setup. Now a billion or so people can't acces my server.
Rate limits didn't work because they kept rotating IP addresses.
I'm pretty sure Turnstyle would allow more people through than my current solution, but this was quick and easy. I expect to have to ban more ASNs from other countries in the future but the worst bots are now gone.
Those images also used to crash all the early GUI irc and chat clients that showed inline images without size checks...