Is there a way to make a mastodon instance for a private community where it requires registration (& approval) like a forum, but still have the federation between instances? (if not, is it possible to unfederate an instance?)
Yes. It's very easy to lock down registrations, say to your family or coworkers, and still federate. You can also turn off federation if you want an internal-only instance.
As I understand it, you can require users to get approval in order to register locally, and there are at least patches out there to create a whitelist of instances that are allowed to federate with you, but there's no way to require users from the allowed instances to get approval before interacting with local posts.
Private Accounts (also sometimes referred to as Locked Down Accounts) require explicit follow approvals and many federated actions are restricted only to direct followers. It's an easy task if you are an instance admin creating all accounts to default new accounts to Private instead Public (which is the usual default elsewhere). Accounts can switch themselves between Private and Public as needs change (though there are obvious caveats when switching).