Unless you're white and male, and then you'll be accused by the radical fringe as wanting to rape everything you come across and kill everyone not like you.
Consider another explanation: If you use the term "rape culture", you essentially claim that in today's society, oftentimes rape will be accepted (FWIW, I agree with this statement). However, there appears to be a consensus in today's society that genocide is always a bad thing; generally that opinion is expressed without any qualifiers (as opposed to qualifiers like "legitimate rape" etc.)
I definitely think that's a big part of it. A lot of the people involved in these discussions probably have been sexually assaulted, but relatively few have had their lives affected by genocide (or even murder).
Puritanism probably plays a role, too. Even consensual sex is often considered an inappropriate topic, so it wouldn't be surprising if that applied doubly to rape.