[procaryote]

750k items in their content management sysem. N independent labs crawling wanting to check that every day could easily give bursts of millions per hour

Millions per hour is tens per second though; perhaps the fix is performance improvements

[setupminimal]

We have put in a number of performance improvements, yes. The nice thing about those is that they also make the site snappier when it's not under load. Right now we have just over a million items in our CMS, plus our publicly available mailing list archives, which are much larger, even if they're less frequently referenced.

[andai]

Yeah we can just rewrite the web servers in Rust ;)

That'll be great until.. they rewrite the scrapers in Rust! Then we're really hosed!

[Jabbles]

1,000,000 / 3,600 = ~278