No reason to be concerned as long as local laws are being followed, which it sounds like is the case. Below 400ft above ground is fair game for everyone, as normal air traffic does not fly that low, making it perfectly safe.
Perhaps I'm wrong to frame it as a privacy issue, but I have a large problem with allowing people to be harassed for lawful actions. If the law says I can do something, and I do it, I shouldn't be required to explain why I did it.
I have no problem with people not wanting drones flying over their properties, and I have no problem with a collective decision that people's privacy should supersede the right to fly drones in public airspace (indeed, I'd probably agree with it). But that should be enforced by a change in law, not by an demand that someone justify their supposedly legal actions.
Well unless you care about privacy, the potential for stuff crashing on you/your property, and/or noise pollution.