If you don't want to trust anyone but to verify instead, consider running Matrix with your own server. In this case you still can talk to anyone else on Matrix, because it' federated.
I'm not familiar enough with Matrix, but I'm curious about how/if it can protect against malicious federated server operators? Does that just boil down to needing to trust they are not running modified code?
Your own Matrix server does not spread any information unless you allow it or unless you message other servers.
Now, concerning the other servers, it may be a problem, just like Gmail is a problem for everyone running their own email server. However, it's a much smaller problem and at least theoretically everyone who cares can escape the walled garden.
You can create a server for all your friends and you will always know exactly which information is shared with others and which isn't.