[tptacek]

What's the message format you're using for RSA? I see RSA in the "pq" code of your Java client library; I see you asking Java for an "unpadded" instance of the RSA encryptor; I see you using RSA in its encrypting operation (not signing).

Great crypto software avoids RSA. Good crypto software uses OAEP when RSA is used to encrypt. Crappy crypto software (see: DNSSEC) uses PKCS1v1.5. What are you using?

I didn't look carefully; I have probably missed something. Please explain how you are using RSA in the handshake/authentication for your protocol. Like Moxie said: it's not the underlying algorithms you use; it's the constructions you use to to them together, and to the data you're protecting.

Incidentally, "I refuse to respond to comments from people who have only looked at my system for a couple minutes" is not a habit of highly effective cryptographers.

[TelegramApp]

This version of the client is using manual padding with random bytes. This isn't too bad, considering the SHA1 presence. (See #4 here: https://core.telegram.org/mtproto/auth_key)

The server now supports OAEP, although this has yet to find its way into the docs and clients. We will update. Thank you!

[tptacek]

This is a brand new application, right? Why is it built on '90s (or, worse, '70s) cryptography?

You need to publish the server-side source code for me to have any chance of making sense of this. I only see the client side of the connection in your source code, not the server response.