Yes, but it should be noted identity servers are only used for associating phone numbers and email addresses to Matrix IDs. Matrix IDs themselves are federated just like email addresses (@cyphar:cyphar.com is mine, and it's hosted on my own homeserver at cyphar.com).
If you don't register your phone number or email address (which last I checked is not part of the default account creation flow) then it really makes no difference. I agree they should be federated (and they have been working on that among many other things) but it's not like every user's identity is centralised.
Optional security isn't security. Especially with the sort of metadata that is in _someone else's phone_. Basically, everybody who has my phone number probably has it in their iOS/Android contacts. I can't opt out of _them_ using a bad identity server.
But you already can't opt out of them sharing their entire address book with the latest sketchy app they downloaded. What's the difference?
They either have your phone number and other contact details in their phone or they don't. They either make good decisions or they don't. You choose how much to trust them and what with. Federation and third party implementations of identity servers for one particular app changes absolutely none of that.
If you don't register your phone number or email address (which last I checked is not part of the default account creation flow) then it really makes no difference. I agree they should be federated (and they have been working on that among many other things) but it's not like every user's identity is centralised.