I don't think Exchange uses any interface other than LDAP to hit Active Directory, so I'd think there's at least a fighting chance that the AD requirement could be satisfied by other LDAP servers.
I can't speak to the IIS integration anymore-- I quit following that at a low level back in the Exchange '03 timeframe. I would suspect there's a clear line between IIS and Exchange, however, that could be used to facilitate an adapter layer for a flexible HTTP(S) server like nginx. The SMTP, POP, and IMAP services, though, would probably involve a lot of re-implementation.
I can't speak to the IIS integration anymore-- I quit following that at a low level back in the Exchange '03 timeframe. I would suspect there's a clear line between IIS and Exchange, however, that could be used to facilitate an adapter layer for a flexible HTTP(S) server like nginx. The SMTP, POP, and IMAP services, though, would probably involve a lot of re-implementation.