Closed IM protocols aren’t standards so this doesn’t really apply. There’s only three realistic “standards” in instant messaging: irc, xmpp and matrix. IRC is essentially dead, xmpp is on its last leg and mostly irrelevant, and matrix is very promising just not popular enough yet.
XMPP is widely deployed and it's the backbone of WhatsApp, you don't see a lot of servers that openly federate XMPP because there's little business incentive to do so.
Xmpp can continue being a server side protocol all you want, it doesn’t make it a user facing protocol. So in terms of federated or at least cross compatible IM clients it’s irrelevant.
A mere protocol will never "hit the bigtime" without some compelling (non-protocol) reason. It's unfortunate that businesses choose to build silos time and time again, but that's where we are - it's not an issue that can be solved at the protocol level.
Should this be fixed (e.g. through regulation), XMPP is absolutely a sound choice - given, as you say, its 20 years of experience, evolution and deployment track record.
Having hardly anyone switch to a new standard when there are existing ones could actually be considered a good thing. The problem comes when the new standard is promoted, inevitably fails to become dominant and ends up making the problem worse.
Note that sometimes new standards are promoted specifically to destroy an existing open standard. The trick is to push hard for the new standard until it gets sort of popular and then back off. Adjust as required to keep things balanced.