[arendtio]

Well, there are different ways. If you want to know what your client is sending and you use your own server, then your server might be able to show you what it receives. ejabberd for example, lets admins inspect the stored offline messages. That way it is very easy to see if a message is encrypted or not. But you could also run a MitM proxy.

Another option is trusting audits or the developers. Last but not least you can inspect the source code of open source apps. So I don't know how deep you want to go with this, but for XMPP there are plenty of options to make sure the client does what it advertises.

Btw. I do not think that OMEMO is fundamentally better than WhatsApp does, as they are implementing the same protocol (Double Ratchet). The main differences are that one is an experimental public standard while the other is a proprietary protocol extension.