There's nothing punitive about saying "this doesn't work for me". Sometimes, it's not even about the other party.