[davexunit]

You can't. WhatsApp is proprietary software that you aren't allowed to audit. Use free software chat programs instead.

[switch007]

How can I (developer, no security knowledge) verify that <open_source_alternative> does encryption properly?

[tptacek]

You should assume it can't. The track record is very, very poor.

[srslack]

That's a funny statement, considering PGP and OTR is the go-to and has held up for all of these years.

[tptacek]

What's the next example?

[srslack]

Truecrypt, also confirmed by NSA documents published by Der Spiegel to be 'catastrophic.'

OpenVPN. SSH-2 with RSA keys.

What proprietary software with good track records did you have in mind?

[tptacek]

Truecrypt isn't a messaging system, is barely open source, and is barely trusted (though I think that's unfair). Compare, on the other hand, to "real" open-source disk encryption projects like EncFS/Ecryptfs.

OpenVPN is built on OpenSSL and was Heartbleedable.

Until a few years ago, SSH was a fiasco. Cryptographically, it has approximately the same security track record as SSL. It's also not a messaging system.

I didn't say I had a closed-source alternative for you. There aren't good answers here. I like TextSecure. I also like GPG, a lot. And I have a 4-figure bet with Matthew Green that OTR is more resilient than the other messaging systems. But OTR is mostly only OK if you don't use it with an actual chat client; once libpurple is in the picture, nothing is OK anymore.

[JoshTriplett]

Look for reviews by reputable security experts. Look for use of a standard cryptographic library, preferably a high-level hard-to-get-it-wrong library, rather than hand-rolled cryptography.