It is similiar to Mastodon in that most users (from my impression) still sign up at a main host, or instance, trusting that their sign-up data is secure (the important thing are keys which are generated upon sign-up). Once that initial account is created though, one can start using any number of other hosts which can also act as clients. This would be one difference - with Mastodon, your account made at ABC instance stays with that instance. Another similarity though is that anyone can make their own instance, generate their own users keys, and then start using any host/client they want, including obviously their own. So, federation (but federation within the nostr world only) happens mostly through the freedom of users to move about, and by users being able to make their own instances (which then have to have "relaying" turned on to help federate). Nostr though also has a way for users who've made their own instance to also turn it into a node which can receive sats as payment for being a node, and this aspect is often promoted in whatever article you find about Nostr, which has, in my opinion, the unintended side effect of scaring people away - partly because it is quite technical, and partly because it's scary crypto.
Great question, and I'll let someone with deeper technical understanding answer this, but using Wikipedia as a reference (1), it looks like "fediverse" is a bit loosely defined, with parts of it not being fully compatible with all of the other networks (2).