[seldo]

CDNs don't usually cache 404s. VSCode was looking for @types packages for any and every npm package its users were using. Packages that had a type description caused no issue, but most packages don't, so we had a > 1000% spike in 404s. Our workaround before MS did the rollback was to cache 404s for @types packages specifically, and it was effective enough that the registry never really went down.

[akfish]

Interesting. Thanks for sharing this information.

[tankenmate]

It's a pity, DNS handles negative lookup caching / TTLs (in fact that is exactly what the TTL field in DNS zones are!). So negative lookups can be cached, but you need to do thinking about it ahead of time and set sensible TTLs (preferably configurable) for those negative caches.

[natuac]

"a > 1000% spike in 404s" overloaded your servers? Such are your generation times? Can I bring the entire NPM ecosystem down from my ADSL line using some silly threaded code to make requests to randomly named packages?

[seldo]

99.9% of our requests are handled by the CDN. The CDN doesn't cache 404s, so 404s are handled by our origin servers, which are much fewer in number and therefore quite easy to overwhelm.

You're right that our handling of 404s was naive, and that's definitely something we'll be improving as a result of what we've learned from this incident.