Maybe you'll understand it when I replace one inalienable right with another in that sentence:
"I don't get how it makes me worse off if someone else voluntarily consents to selling themselves to slavery?"
It is much easier for your government to institute a repressive regime if most of your fellow citizens have given up their privacy. Once instituted, the regime can prevent you from leaving the country and harm you in many ways even if you personally were very careful to preserve your privacy.