The problem is that it's a rather confusing Alt Mode.
Basically, it's only supposed to be used for USB-C-to-3.5mm adapters, and it was only implemented by some smartphones. It's not a reliable way to get audio out of a USB-C port, and because it's using the internal DAC the audio quality isn't going to be good anyways.
The proper solution is to use a USB DAC. This provides the same user experience but it actually works with all USB-C devices. Considering the official Apple USB DAC is available for $9, the Audio Alt Mode really doesn't have a reason to exist.
Why is it confusing or bad? Sure, it sacrifices a data line pair, but the same can be said of DisplayPort Alt Mode - if you want 4K@60Hz on DP 1.2 you need to sacrifice USB3.0.
The ability to make this tradeoff should be up to the user.
Basically, it's only supposed to be used for USB-C-to-3.5mm adapters, and it was only implemented by some smartphones. It's not a reliable way to get audio out of a USB-C port, and because it's using the internal DAC the audio quality isn't going to be good anyways.
The proper solution is to use a USB DAC. This provides the same user experience but it actually works with all USB-C devices. Considering the official Apple USB DAC is available for $9, the Audio Alt Mode really doesn't have a reason to exist.