There's subtle, dangerous ways this can be exploited. (Short version: It'd make SRI usable as an oracle to confirm or deny guesses for the content of a cross-domain resource.)
Couldn't this mitigated by user-agents introducing random, Poisson distributed delays in all cached responses? The peak of the distribution could be made user configurable to make it further difficult to predict a user-agent.
It leaks private user info -- a malicious server could include a JS file confirmed to be highly sensitive/top secret, and measure whether the client already has that cached. If so then the user is confirmed a sensitive target.
No, there's a worse attack possible: you can attempt to include a resource with sensitive contents with SRI, and use the SRI to make a "guess" at the hash of the contents. If your guess is incorrect, the resource will fail to load, and you can detect this error and make another guess.
Obviously, this technique will only work if the contents of that resource are constrained enough that it's possible to guess them with brute force. Depending on how SRI interacts with the browser cache, though, it may be possible to make guesses very quickly -- it is likely that the browser will only fire one HTTP request for the initial attempt, and will load the resource from cache for all subsequent attempts.
you could just add a new public=true option to counter this. I think you can even already check that with an iframe (or js head inject & timing) anyway, no need for CSP for that.
So to protect against a single malicious server who might discover that we had previously loaded a cached resource, we shouldn't implement a cross-origin cache and have to make repeated requests, guaranteeing 3rd parties (the CDN) keep getting GET requests?
You're just trading one problem (someone learning I previously requested a file) for another (leaking referrers to a CDN).
Also, if you're loading "highly sensitive/top secret" data with a <link integrity="" href=""> or <script integrity="" src=""> tag, you have bigger problems.