Easy, each library is its own little world of dependencies, with the author favoring a different build tool, so simple npm install or npm serve might require all of them to satisfy all the dependencies build scripts.
I think you're confusing something. You don't "build" dependencies, you consume them as pre-built modules.
As a package author you use whatever you want (rollup, webpack, gulp, grunt, ...), but you will most likely publish to npm, because bower is dead.
As a consumer you use npm or yarn, it doesn't matter, and a tool that handles modules, probably webpack. Even if you'd use something else (rollup & parcel come to mind), it doesn't matter, they will all treat your dependency the same.
EDIT: Yeah i just saw you're referring to Polymer. I take it back, everything you said applies. But then again, people probably want to suffer when they use it. Though very few actually do use it. I just don't think it's a good example to make to describe web dev in general. Rather Google was the only one that missed the train it seems, they worked in the opposite direction.
Polymer is indeed ridiculous. But there's a reason it's not used that often.
My relatively standard project (using React) needs just Yarn and Webpack. It's got quite some dependencies, but I've never run into those being insufficient.