Most people already have a phone, laptop, maybe a watch, maybe the TV remote.. And lots of apps on each one. Any one of which could be listening in. It's a crazy situation.
Android has a button in the quick settings bar to enable/disable the microphone, which helps with this (as long as you trust the OS itself). I keep it disabled most of the time.