The spam filter can be customized, but I agree that it isn’t nearly as good as gmail’s. I have it configured to move messages with a score higher than 3.5 to spam which prevents all false positives for me, but 1 or 2 spam emails a week still get to my inbox. https://www.fastmail.com/help/technical/spamchecks.html#cust...
I agree that the comment manager is barebones. Fastmail does support CardDAV so you could use a different contact manager but I haven’t had the need.
Edit: I swear I am not a Fastmail shill. Just a happy customer!