IIRC IME also does a lot of core functionality like power regulation. Unlike many in this thread probably think, it does provide a lot of core functionality that you probably don't want removed.
The contention, from the User standpoint, however, is the network stack, potential to phone home, and the unrestricted access to the global machine state, combined with the fact, it is not documented or disclosed.
It's one thing to have that and be up front and open on it. Get secretive, and you're creating a massive source of unknown unknowns for everyone involved.
And like it or not, if you won't/can't be transparent about it, either
A) It'd take too long to document, which suggests there may be room for simplification
B) you're doing something that if it saw the light of day, would cause outrage, likely because you shouldn't be doing it
C) You're holding back the state-of-the-art for the sake of securing a revenue stream.
None of these inspires a excess of confidence/trust.
Oh, great! Isn't there a way where intel could provide keys so we could get rid of IME even if it means we won't be able to play DRM'ed content?