But the intent isn't there for Amazon to listen to everything you say - which is why hotword detection is done locally on Alexa and only then is an audio stream sent to their servers.
I have a coworker who has Siri enabled on his phone. It regularly wakes up and starts recording seemingly at random. Maybe "always listening" isn't the correct phrasing. "It might be listening at any time" might be better.