[Cthulhu_]

The community response was respecting robots.txt, but since that was more of a gentleman's agreement than a legal requirement, hungry AI scrapers disregarded it.

Which brings us to the old fashioned flood control mechanisms. That is, the toolkit you propose already exists and has been used for decades in various iterations to protect against various forms of attack (slashdotting, DDOS attacks, overzealous search engines, and now AI scrapers).

Have you looked into those before? Companies like Cloudflare have been at the forefront of this field for a long time now.