[rogerbinns]

It was more akin to a self signed certificate, but also more complicated depending on domain setup. But also things changed a lot between RDP protocol versions.

It did take me quite a while to reverse engineer the encryption. It was mostly done using IDA Pro and SoftIce. The hardest bit was because I had several crypto books, and whichever Microsoft employee implemented their security kept doing what the books said not to do, so I'd never expect to keep encountering the "wrong" way of doing things.

https://www.hex-rays.com/products/ida/index.shtml

https://en.wikipedia.org/wiki/SoftICE

[yuhong]

http://www.rapid7.com/db/vulnerabilities/windows-rdp-constan...

I think the newer RDP versions uses TLS BTW.

[rogerbinns]

I did the reverse engineering on RDP 4, almost 10 years before that vulnerability report. With each RDP release they did very big changes to the security layer as well as other pieces. Somewhat surprisingly Microsoft did base RDP on open standards, especially a whole bunch around video messaging.

But those standards were developed by committee, so the number of ways you could do things was usually the same as the number of committee members. Then their implementation had bugs (eg drawing some parts from bottom to top when the standards said top to bottom), but they didn't know this since they didn't try interoperating with other implementations. The initial "security" was some half assed hack, and yes in later versions they did switch to TLS.

Note that the security is complicated because of domain membership, WAN vs LAN, and a bunch of other related issues.