No worries with sound. There are plenty of audio enthusiasts that will have nothing to do with a system like this and there will always be plenty of options because of them.
I know this isn't your central point, but you should know that a speaker can be used as a microphone, and vice versa. Granted, a speaker won't be a very good microphone, but it is possible one could design an exploit that uses this capability to record audio and send the results off to who knows where.
It's unlikely that a basic Bluetooth speaker would even have an ADC to switch to, not to mention speakers without Chromecast-like streaming capabilities probably only have some tiny real-time operating system.