[colanderman]

I suppose, if they published the algorithm they claim to use, and if you can stub out the app's random number source (say LD_PRELOAD or the like), and if you can sniff the app's network traffic (again, LD_PRELOAD might be necessary if it's encrypted, assuming they're not using a statically linked SSL library), and if they don't perform one of any number of trivial modifications to the algorithm (such as adding a fixed salt), you might be able to, for a given message, confirm that for that message, they encrypt it identically to how the algorithm they claim to use would encrypt it.

But that's a lot of ifs, and doesn't prove there's no backdoor that's currently disabled.