You're assuming that you'll even be able to tell who's wearing one rather than regular glasses. Look at the change in form factors of MP3 players and phones over the past 10 years, and then look at where they're _starting_ from with these. No reason they couldn't be nearly invisible by the time they hit widespread use.
You'll be able to spot those people because they're always walking into things or muttering voice commands to nobody in particular.
Don't tell me you can't spot someone with a portable music player from a hundred feet away. If anything the visibility of those has increased as the popularity of over-sized headphones has grown.