Why worry about something that hasn't happened? Why not worry about a careless teenager wanting to get a sick Camera angle of the inside of your fridge? It's as valid a concern.
Normally I would agree with you, I often chide people for inventing problems to solve. But it is a little different when it isn't a hypothetical as much as an inevitable future occurrence.
As for refrigerators, I am just as worried about people trying to fly drones around my house as I am about people flying drones around planes, yes. I have it on good authority that anyone with the funds can purchase access to 3D drone-generated map of my city, my house included. A gentleman here in town who replaces windows doesn't bother to measure them anymore; he just uses the drone photos.
We will always have to worry about intelligent, motivated people who wish to do us harm. Isn't that enough to worry about without being vulnerable to careless people, behaving unintelligently, who put lives at risk through their ignorance? Keep in mind, the latter group is much bigger.