[ocdtrekkie]

This seems largely practical/beneficial, and not of a significant downside. If you are pointing your domain at them for DNS, they are arguably canonically "that server" until you sent it somewhere else, and they're securing the connection for you.

The short-term nature of Let's Encrypt also works out well for this, because if you take your domain elsewhere, FastMail loses the certificate to claim to be that domain very quickly due to rapid expiration.