People who attend conferences about women aren't a protected class. So it wouldn't be illegal unless you could demonstrate that filtering out people who attend conferences about women systematically discriminated against women. This would be fairly difficult at best, and almost impossible unless it were a large organization with an obvious gender imbalance.