[crispyporkbites]

You already know the answer to this. The context of your fact-based statement is where it's meaning (and any offence/labeling) is derived from, not the fact itself, and in this case the person you are responding to is adding an additional fact without citation:

This is the "simple" fact:

> Women placed compensation at 4th in "Differences in Assessing Jobs by Gender" in a StackOverflow survey targeted at people (mostly developers) who use stackoverflow

This is the additional fact that has been added:

> [that fact is] a contributing factor for the pay gap

And then the phrase

> you can be labeled as sexist

is added, which of course can be applied to anything anyone ever says about anything gender related.

If you deliberately add unsubstantiated information to a fact you will probably be called out for it, and rightly so (IMO).