It's also possible it's part of a docker build step or similar. Even if they're aren't downloading models at run time they may be loading s3 if their pytorch-transformers lib docker cache gets invalidated frequently.
Docker builds are crazy wasteful in terms of bandwidth and compute. Right now I'm struggling with a project that builds ITK on demand every heckin time.
I'm working through how to best integrate apt-cacher-ng, sccache, and a pip cacher. It costs me nothing to hit apt or pypi, but like, somebody is paying that bill. A little perspective goes a long way.
I wonder if I could do something to just proxy all requests and cache those on a whitelist and stick it on my CI network.
I'm working through how to best integrate apt-cacher-ng, sccache, and a pip cacher. It costs me nothing to hit apt or pypi, but like, somebody is paying that bill. A little perspective goes a long way.
I wonder if I could do something to just proxy all requests and cache those on a whitelist and stick it on my CI network.