That is very true and a mistake I do make pretty often but am working to be more cognizant of. Framing the discussion around women and men does nothing to further those who are identify as both or neither, nor those who are in transition from one binary gender to the other.
Can everyone see what just happened here? This is all it has to be. It was a cis-sexist remark. Now, I tend to make similar remarks and I also try to be aware of them, and there's indeed a limit to what you can do. But this is _all we're talking about_. OP was being facetious, but a subtle '-ism' was pointed out, and the commenter just said, "Oops, yeah, working on that." And we all happily move on with our lives.