Do you have a sound bar or external speakers? If so, I assume you only have one of your devices connected to them? With USB-B the audio source is determined by whatever device is currently selected by the monitor. It's nice.
I used to use a cheap mixer[1] but recently I found some speakers with multiple inputs. It would be great if everything could just route through the monitor instead.