Yes, the selfish-minded would end up with more selfish-minded people, and they'd be confused why their "low trust society" became even more low trust overnight.
Confusingly, though, as you are of course a nice person, if you vote red you'd demonstrate that some red voters are nice, and then the choice is less severe. Then voting red is like "I embrace humanity, warts and all", while voting blue is like "I cannot tolerate sharing the planet with anyone even slightly impure".
"Empathy" isn't a binary in this context. You can exercise empathy and aid your community by making sure everyone you know votes red. That's the kind of cooperation that humans have evolved with. What you're talking about is undifferentiated, universal empathy, where someone would be willing to risk the lives of those close to them for a greater chance to help those who are outside their immediate reach to persuade.
I suspect if you played this game, lots of tight-knit, high-cooperation groups would undertake coordinated campaigns to ensure the survival of their members by ensuring everyone votes red.
Well that's exactly how empathy plays out in real life. It's not an abstract feeling. People often put their lives at stake for others, which is something game theorists can't really appreciate.