That's not the real question, you're pretending context doesn't matter. What's your argument? Women say "I enjoy having a place to communicate with other women in a place that feels safe from male interference," and your response is "Let me into the women's bathroom"?
No, having separate bathrooms is not sexist. Because the term sexism refers to negative issues only. Webster's uses the word "discrimination". Which I assume you are being pedantic and interpreting in a broader context like "discriminating between the colors blue and teal".
But lookup discriminate and it has a few meanings. One of which refers to prejudicial or unfair treatment. That is the definition implied when referring sexist discrimination.
Every major dictionary specifically uses terms like "prejudice", "discrimination against", "stereotype". There is obviously a consensus that it is referring to negative/unequal treatment.