The difference is that because Xbox One feels like it's watching you, people's minds jump to how people could end up watching you, even though the actual risk is far less than with the devices you mentioned.
Why does it feel like it's watching you? The Moto X is constantly listening to you. The PS4 has a camera as well, along with every smartphone. What's different about the Xbox?