Thing is, we can build better buildings that are more resistant to noise pollution, regardless of the source. If I can't hear my neighbors, then I can totally ignore them. Which makes it livable.
The problem with this is that it reinforces the notion that you "live" inside and only go outside when you need to get somewhere. How are you ever going to enjoy a nice street-side cup of coffee in peace? Or take a walk around your neighborhood? Or engage in any outdoor social activity?
If you allow the outside to be unpleasant, then people are increasingly going to stay inside which has negative societal ramifications.
I think we can agree that not being woken up by your neighbors drunkenly blasting music at 3am is preferable to the alternative. But people should be able to live their lives and if they want to celebrate a big win at work or whatever, as long as it doesn't bother me, who am I to say they shouldn't at 3am on a Wednesday? Yes the outside needs to be nice as well, but we have to live together, and the best way is for me to be unable to hear them.
The outside needs to be nice, but making inside unpleasant doesn't accomplish that.