[UnoriginalGuy]

First off: Totally true.

Secondly: Devil's advocate, but it is a "hard problem." It is easy to look for behaviour on the system, it is very hard to look for patterns of behaviour.

I mean let's say that some of your users are normal court clerks, it wouldn't be unusual to see them sit around and pull tons of records all day every day. So how do you pick up normal requests on-mass and unusual requests on-mass?

If I was in charge of protecting such a system I wouldn't even attempt to detect this (too hard). Instead what I would do is make it impossible to get records sequentially (e.g. 1, 2, 3...9999999) instead each record had a unique randomly generated token associated with it (a UUID/GUID).

So in order for someone to gain every single record they would either need to conduct a "real" break in and steal the files, or search for every possible criteria (which, for them, becomes a huge hassle/problem).

PS - Most DDoS are, these days, against layer 3 (network). Since it is far harder to defeat a layer 3 attack (as it can literally crash a lot of network hardware). While layer 7 (software) DDoS attacks still exist, they're often conducted by less formidable adversaries and they're much easier to stop (e.g. return a JavaScript redirect instead of the normal page, most browser-users won't notice, but it will defeat a targeted attack until they re-target (and you could rename it every 10 minutes)).

[dredmorbius]

So, here's a story I heard recently.

The person involved wanted to create a local archive of records. An index of material was possible to obtain, but rapid sequential requests resulted in an IP block preventing further access.

Modest levels of restructuring the requests, in random sequence, with a significant (several minutes) delay between requests, and random delay, eventually succeeded in retrieving the material.

If that had failed, a distributed set of requests could have been attempted.

When I've faced issues of high (to the level of service-degrading) levels of traffic, I've found tools that allow me to aggregate requests by similar attributes, including requests coming from a defined network space (CIDR or ASN), which can be quite useful. Reading such patterns just from eyeball scans of logs is pretty bloody difficult, and tools to assist in this are ... poorly developed.

[meowface]

>Reading such patterns just from eyeball scans of logs is pretty bloody difficult, and tools to assist in this are ... poorly developed.

There's some enterprise software out there designed for use cases like this, but they're typically very expensive. There are also other issues, like the storage requirements of full logging of request headers and bodies if you really want to see the big picture.

Simple IP rate limiting will stop the majority of would-be scrapers/scanners in their tracks though. Especially if there's so much material that it could take days or weeks to finish a scrape if you had to add a random delay of 3 or more minutes per request.

[meowface]

I do network security for a large company, so I'm not talking completely out of my ass when I say you can have alerting in place to at least detect the most obvious behavior. There are also tools and even entirely inline appliances (look at RSA Silvertail) designed specifically to look for automated behavior against a web server.

Someone clever enough will be able to get around it, but it's really not hard to detect automated scanning or scraping behavior, especially if they're not delaying their requests in any way.

Stopping a layer 3/4 DDoS is another matter entirely. They're quite easy to detect but quite hard to mitigate yourself; you need your upstream provider to mitigate it for you.

Also, using Javascript interstitials against layer 7 attacks (like Cloudflare and Incapsula do in their default mode) will stop script kiddies, but they're not hard to get around if you know what you're doing. So you'd either have to, as you say, change the method every few minutes...or just use a captcha.