The funny thing about human rights is that one can never justifying denying them because the minority who wants them is too small to count as important.
Indeed, the number of people being tortured at Guantanamo were a tiny, tiny portion of the global population - doesn't mean they should have been subjected to that, regardless of what they were accused of.