[chrissnell]

I dealt with this problem at a previous job where we had a build pipeline and apps that were very dependent on NPM. To fix it, I used nginx to build a set of two-tiered caching servers within our CI Kubernetes cluster. One used a ramdisk to provide a low latency cache for the NPM client fetches. The second was a disk-backed cache to provide persistence to the ramdisk-backed cache. I had a script that would alter the package deps so that they were pulled from the ramdisk-backed service, which used the disk-backed service as it's backend. The disk-backed service uses actual NPM URLs for its backend.

The result was lightning-fast fetches and no rate limiting.

We needed the two-tiered system because this was Kube and occasionally we would have to rebuild/restart nodes and we didn't want to completely lose the cache when that happened.

You can easily extend this system to handle any package artifacts used in your build process: .deb's, .rpm's, etc.