Something similar could've been said about both interracial and same-sex marriage.
If the relationship type (interracial, same-sex, poly) is considered in some manner deviant or otherwise unacceptable, the number of people openly engaging in it will be small. As race relations in the US improved, as attitudes on same-sex relationships improved, the number of people openly engaged in these relationship types increased.
As poly relationships become more acceptable, more people will admit to being in them. Then we can examine what percentage is impacted by not being able to marry.
The funny thing about human rights is that one can never justifying denying them because the minority who wants them is too small to count as important.
Indeed, the number of people being tortured at Guantanamo were a tiny, tiny portion of the global population - doesn't mean they should have been subjected to that, regardless of what they were accused of.
If the relationship type (interracial, same-sex, poly) is considered in some manner deviant or otherwise unacceptable, the number of people openly engaging in it will be small. As race relations in the US improved, as attitudes on same-sex relationships improved, the number of people openly engaged in these relationship types increased.
As poly relationships become more acceptable, more people will admit to being in them. Then we can examine what percentage is impacted by not being able to marry.