Apologies if I'm being dense, but is there some technical reason why it does not use the referrer to selectively block this traffic rather than blocking it for everyone?
Sure. I suppose that could be the reason. However, something like HA proxy[1] and (IIRC) CDNs like Akamai can be configured to inspect the headers and take various action.