And if the federated server you are on goes down. You lose everything. If the federated server you are on steals your information or censors you there is nothing you can do.
You have to choose a server that you trust. In centralized systems, you have no choice at all. If you can't find a trusted server, you can set up your own, ask a friend to do it for you, or pay someone to do this job. You are not bound to an artificial monopoly.
A friend and I have been running a private Matrix server for almost a decade now, it's very lacking in comparison to what the average chat user (especially discord) is used to.
No custom emojis, no self-chat, embeds are inconsistent (e.g. encrypted rooms), multi-image uploads aren't a thing in many clients, adding text when sending an attachment isn't a thing, just to name things we've run into over the years. Most of these have been brought up to the devs many years ago, only to spend forever in spec hell and never actually make it into a release.
We're just tolerating these, because we explicitly moved off discord to have control over our data, but being tech savvy we can handle this. It's nowhere near good enough that I could use it with less savvy people.
Everything about matrix is cumbersome and glitchy. I have last tried to use it a few years ago and it seemed that Riot/Element had the only decent clients, and those were all Electron on desktop and also seemingly for profit. Signal has the electron problem, as well as many others (like the backup UI being abhorrent), but at least the core functionality works without fuss.