[dgacmu]

I had meta's crawler hitting the pi searcher at something like 5qps for days on end, just ... querying for substrings of pi, ignoring robots.txt, etc. it wasn't enough to break anything but it triggered a lot of alerts.

I can imagine that sites with dynamic content and potentially unbounded query types or pathnames are in danger from particularly stupid crawlers.

[leros]

Meta has hit me with over 1,000 requests per second. Luckily it tripped my rate limiting but geez.