[meowface]

>Reading such patterns just from eyeball scans of logs is pretty bloody difficult, and tools to assist in this are ... poorly developed.

There's some enterprise software out there designed for use cases like this, but they're typically very expensive. There are also other issues, like the storage requirements of full logging of request headers and bodies if you really want to see the big picture.

Simple IP rate limiting will stop the majority of would-be scrapers/scanners in their tracks though. Especially if there's so much material that it could take days or weeks to finish a scrape if you had to add a random delay of 3 or more minutes per request.