Also, issuing servers and verifying servers don't even need to be part of the same organization, allowing you to outsource credential management (see Auth0, Firebase Authentication).
By downloading a shared key over TLS rather than the provider's public key?
No difference from the perspective of the token consumer. From the perspective of they token generator, it means rotating per-tenant keys rather than a single keypair.
I addressed this elsewhere (https://news.ycombinator.com/item?id=16072690) but to quickly recap: that's not the hard problem, and hardened SAML IdPs that have the option of exploiting this turn out to have per-tenant keys anyway so that they can get cryptographic binding instead of counting on audience restrictions being checked.
Additionally, your TLS terminating stack is much better hardened than median in-app crypto code.