[wfunction]

> There are more points to consider than that. For instance, which order should you apply the ciphers in?

No. The order shouldn't make any difference unless for some reason you're sending extra data in cleartext that is encrypted with one cipher but not the other. This is because the output of the standard cipher (e.g. AES) would look random, so that implies the final output must look random, and hence they won't be able to tell there's another layer on top just based on the order of the ciphers. That is, unless they've already broken the other standard cipher (in which case now you're only dealing with the custom layer regardless). If the final output isn't random, it means you're partially reversing the standard crypto, which, as I said above, cannot happen unless you've broken the crypto or avoided using independent keys.

Edit: I suppose the theoretically optimal thing to do may be to apply the standard cipher last, to absolutely, positively ensure that the adversary is forced to break that before they even know you have another layer underneath (to avoid parallelizability of breaking both). I can't imagine this ever being worse. But at this point we're talking about theoretical optimality; from a practical standpoint I don't see this mattering. But at the same time since I don't have an argument for doing it the other way, you might as well always do it this way.

[im3w1l]

I would suggest the standard cipher first. Because, if the home brewed one is used first, it may leak information over side channels.

Another concern is that if the home made cipher creates a cipher text with differing lengths depending on the content of the plain text, the standard cipher will not be able to obscure that length.

[wfunction]

> I would suggest the standard cipher first. Because, if the home brewed one is used first, it may leak information over side channels.

Ahh! I remember realizing this once but then I completely forgot about it. It's a good point, thanks for mentioning it. The thing to note here though is that the only side-channel attack here is the time taken for the encryption to occur, since we're talking about networks (and not physical penetration of the system's environment)... which is admittedly nontrivial to defend against with modern CPUs, but which is not quite as hard to do as it might seem, if by side channel people think of the same thing I normally do (e.g. E/M waves from the monitor or something).

So maybe apply a standard layer initially, add your custom layers, then top it off with another standard layer?

> Another concern is that if the home made cipher creates a cipher text with differing lengths depending on the content of the plain text

I guess I assumed it was obvious you would never do this because it's common sense if you know even basic cryptography, and as far as I know, this is literally the only possible failure mode with regards to information leakage in the ciphertext itself, so it's not like you have to worry about other similar situations either. (But do correct me if I'm wrong.)