You are not being punished even if you perceive as such. One could say that an obese women that loses weight and gets in shaped gets punished because men hit on her.
If that hypothetical woman was getting hit by men after she made it clear that she dislikes it, then I'd say it is punishment. Well, not punishment exactly, because that's a response to a behavior considered wrong, but in a "no good deed goes unpunished" sense.