You appear to also realise that if it's a closed source client then the server could be fine, the client could do all the snooping and pass data in a side channel. It's worth spelling that out IMO.
- if it's an open-source client but it doesn't display fingerprints and you haven't modified it, you're stuck. (At least you know you're stuck, but you knew that already.)
- if it's an open-source client but you're trusting someone else's binary, they can attack you.
- if it's an open-source client but you're not trusting someone else's binary, you're not on the embargo list and so responsibly-disclosed bugs are effectively zero-days for you.
- if it's an open-source client but it's written in C, you have no practical way of auditing it against intentionally-malicious source code (i.e., for almost everyone, the cost of auditing it is higher than the cost of visiting your conversation partner in person).