Apologies if I'm being dense, but is there some technical reason why it does not use the referrer to selectively block this traffic rather than blocking it for everyone?
Sure. I suppose that could be the reason. However, something like HA proxy[1] and (IIRC) CDNs like Akamai can be configured to inspect the headers and take various action.
E.g.:
- This works: https://github.com/greatfire
- This doesn't: https://github.com/greatfire/
So far as I know, most links (external or internal) to a Github user's page do not use the trailing slash, so the effects would presumably be minimal.