I don't know, sounds quite similar to his rate distortion theorem (analyzing minimum number of bits/symbol you need to stay under some fixed amount of distortion). I.e. lossy compression with a maximum amount of loss. I.e. "pretty much lossless" compression.
"Pretty much" doing a lot of work. But it's kinda analogous to 99% JPEG compression: yes you can detect loss, but you get meaningful compression ratios out of it and the subjective appearance is nigh-on perfect.
Shannon would be pointing out that if you can throw away half the model without apparent degradation, we're nowhere near packing in all the information we could in training. There must be a better arrangement than we've currently got.
https://en.wikipedia.org/wiki/Rate%E2%80%93distortion_theory