Indeed. When I'm wearing heels and a floor length flower pattern frock I would say I'm just a bloke wearing a dress, but if you insist that's cross-dressing I shan't disagree.
However my gender remains resolutely unaffected and I have no time for people who expect otherwise. Judge a book by a cover you might, but slipping the dust jacket from Moby Dick onto a copy of The Princess Bride certainly won't make it into a story about whaling.
Also, this word always reminds me of Eddie Izzard: "They're not women's clothes, they're my clothes, I bought them".