XMPP just wasn't a very good fit for the problem we needed it to solve. It is a verbose, XML-based protocol (reflecting the time in which it was conceived), and it's a mess of extensions (possibly the designers getting carried away with the possibilities of XML) instead of focusing on solving a single core problem well -- this works against the ideal of federation: what happens when you send a message taking advantage of some extension to a user that doesn't support it, or to a user that does, but through an intermediate that doesn't?
Also, it doesn't tackle spam. The concept that anyone can email anyone would probably have killed email, had it not been largely (if by no means completely) solved -- ironically, this was achieved by all but doing away with federation (email now being a oligopoly of Google, Microsoft and a few more, more or less dictating who gets to send email to whom). Finally, what I suspect was the final nail in the coffin, XMPP didn't play well with mobile (not to worry, an extension, XEP-0286, currently in state "experimental" first published in 2010 is fixing that).
EDIT: I should clarify: XMPP works quite well for private (non-federated) IM-setups. I offer this as an explanation of why XMPP isn't the interoperable email of IM across the Internet, and we instead see Facebook/WhatsApp/Slack/million other incompatible things dominating the space.
The big companies gave up on it. Google, Facebook, etc.
Everyone wants their product to be the new hotness. See the copying of stickers, voice messages, and payments from Chinese/Japanese messaging products. Or how Skype became a Snapchat clone.
Companies giving up on open standards in favor of their own walled gardens does not really provide a meaningful commentary on the open standards themselves.
XMPP itself is a bit of a mess, but it's a standard with a lot of adoption, which has its own value.
A lot of adoption? How many users does XMPP have? I'm glad that XMPP is open, but as a standard, it's useless to me if none of my friends are using services that support it.
Not sure about the actual statistics, but XMPP is not just a standard for messaging between people: I think it's also intended as a backend technology for communication behind automated services. So even though there may not be many humans using XMPP as a messaging application, there may be many people who "use it" without noticing (as in, use a website, etc., which relies on XMPP somewhere).
XMPP Core is a protocol at (roughly) the same level as HTTP: it provides basic semantics for sending and getting data, and you can build your own application level protocols on top of it. XMPP IM is an application level protocol built on top of it that does something specific (instant messaging). It could have been built on HTTP as well, but they were designed to work well together.
If we're comparing XMPP core to HTTP, the difference is that XMPP core is async and stateful (like instant messaging sessions) and HTTP is synchronous and stateless (great for fetching documents).
I don't know how one could get a good estimate of this, but it is still very widely used. Whatsapp (with custom compression and federation disabled), HipChat Server, Cisco Jabber, Android push notifications, Google Cloud Print communications, etc. are all XMPP. But yes, for personal instant messaging Google and Facebook were the big players and they are no longer on it (at least as an interface to their clients)
People mostly stayed in their walled gardens; despite the possibility to leave and explore others. And the companies running chat services decided that it wasn't in their interests to allow people out of those gardens.
I guess if xmpp had been around at the time icq kicked things off things might have been different.
I used to use third party desktop & mobile chat clients for facebook and hangouts, both via XMPP. Then I guess they realized how much user data they were missing out on by not excluding third-party apps and dropped support.
It offers a full social network solution (publication, comments, replies, contacts…) and modern IM (chatrooms, stickers, history management, synchronisation between devices).
All in real-time, multi-platforms and fully built on XMPP :)
Also, it doesn't tackle spam. The concept that anyone can email anyone would probably have killed email, had it not been largely (if by no means completely) solved -- ironically, this was achieved by all but doing away with federation (email now being a oligopoly of Google, Microsoft and a few more, more or less dictating who gets to send email to whom). Finally, what I suspect was the final nail in the coffin, XMPP didn't play well with mobile (not to worry, an extension, XEP-0286, currently in state "experimental" first published in 2010 is fixing that).
EDIT: I should clarify: XMPP works quite well for private (non-federated) IM-setups. I offer this as an explanation of why XMPP isn't the interoperable email of IM across the Internet, and we instead see Facebook/WhatsApp/Slack/million other incompatible things dominating the space.