Revoking consent to a mutually-agreed-upon interaction is not punitive. Consent today does not imply consent tomorrow.
There's nothing punitive about saying "this doesn't work for me". Sometimes, it's not even about the other party.