It's kinda meaningful. Wake words can be sometimes triggered by accident. Or by things like that commercial from a few years ago. A physical button press avoids all that.
So the worry is that it will be triggered accidentally and hear something you didn't intend it to, then that tiny, rarely occurring piece of data will be used against you?