The fact that GitHub don't provide a better solution here has to be actually costing them money with the network usage and extra agent time consumed. Right?
GitHub has perverse incentives to not fix this problem because they charge customers based on usage (by the minute), so they make more money by providing slower builds to end-users.