Having frivolous modules (ala left-pad) brings down the quality of the ecosystem as a whole.
Even if you don't use them directly one of the modules you do include could include them, and they're pretty much only a source of security holes.
NPM in particular could do with much more policing. Between module hijacking, typosquatting, and a lack of security audits of any sort, fewer packages on NPM is a good thing in my opinion.