That's fair, I would agree that caching at either end would fix this. It just strikes me as odd that GitHub, the middle-man that's just providing CI runners, is the one under fire.
What GitHub is effectively doing is providing free DDoS hardware and lots of it, as far as the receiving end is concerned. I don't think GitHub should particularly be "under fire" for this, but it's still very not nice to provide a service that, under legitimate use (never mind illegitimate use!), can make unreasonable amounts of traffic to arbitrary sites.
I think a quite reasonable expectation from GitHub would be to have an all-of-GitHub-wide rate limit that CI can use for requests to any given site, and have jobs fail/delay if GitHub has exceeded that, and expect sites to explicitly opt in if they're fine with more than that rate. Would of course very much suck for GitHub CI users that want to pull from sites not opted in, but at least GitHub would stop offering free DDoS services.