Perhaps I'm wrong to frame it as a privacy issue, but I have a large problem with allowing people to be harassed for lawful actions. If the law says I can do something, and I do it, I shouldn't be required to explain why I did it.
I have no problem with people not wanting drones flying over their properties, and I have no problem with a collective decision that people's privacy should supersede the right to fly drones in public airspace (indeed, I'd probably agree with it). But that should be enforced by a change in law, not by an demand that someone justify their supposedly legal actions.