This is basically the “if you have nothing to hide, what’s the problem if the police can read all your correspondence?” argument applied to an organization.
Sure, if you view government agencies as equivalent to individual citizens and the public as equivalent to the police.
But some people view that the government is properly subordinate to the citizenry and not vice versa, such that inverting the government and public roles materially changes the scenario.
I think that no matter how scrupulous any government organization is, there are many motivated actors who would, given an unfiltered record of everyone single person's correspondence and conversations, be able to spin a misleading negative story out of it. This is essentially some people's full-time job and they're good at it.