[dastx]

Why does everything in node have to have all the dependencies in the world?

``` @sindresorhus/is JSONStream ansi-regex ansi-styles archive-type argparse asn1 async balanced-match base64-js bcrypt-pbkdf bl bluebird brace-expansion buffer buffer-alloc buffer-alloc-unsafe buffer-crc32 buffer-fill buffer-from cacheable-request camelcase caw chalk chownr cliui clone-response color-convert color-name commander concat-map concat-stream config-chain content-disposition core-util-is cross-spawn debug decamelize decode-uri-component decompress decompress-response decompress-tar decompress-tarbz2 decompress-targz decompress-unzip docker-modem dockerode download duplexer3 emoji-regex end-of-stream escape-string-regexp esprima execa ext-list ext-name fd-slicer file-type filename-reserved-regex filenamify find-up from2 fs-constants fs-extra fs-minipass fs.realpath get-caller-file get-proxy get-stream glob got graceful-fs graceful-readlink has-flag has-symbol-support-x has-to-string-tag-x hasbin http-cache-semantics ieee754 inflight inherits ini into-stream invert-kv ip is-fullwidth-code-point is-natural-number is-object is-plain-obj is-port-available is-retry-allowed is-stream isarray isexe isurl js-yaml json-buffer jsonfile jsonparse keyv lcid locate-path lodash lowercase-keys make-dir map-age-cleaner md5-file mem mime-db mimic-fn mimic-response minimatch minimist minipass minizlib mkdirp ms mustache nice-try node-virtualbox normalize-url npm-conf npm-run-path object-assign once os-locale p-cancelable p-defer p-event p-finally p-is-promise p-limit p-locate p-timeout p-try path-exists path-is-absolute path-key pend pify pinkie pinkie-promise prepend-http process-nextick-args progress proto-list pump query-string readable-stream require-directory require-main-filename responselike safe-buffer safer-buffer scp2 seek-bzip semver set-blocking shebang-command shebang-regex signal-exit simple-git sort-keys sort-keys-length split-ca sprintf-js ssh2 ssh2-streams streamsearch strict-uri-encode string-width string_decoder strip-ansi strip-dirs strip-eof strip-outer sudo-prompt supports-color tar tar-fs tar-stream through timed-out to-buffer trim-repeated tunnel-agent tweetnacl typedarray unbzip2-stream universalify url-parse-lax url-to-options util-deprecate uuid which which-module wrap-ansi wrappy xtend y18n yallist yargs yargs-parser yauzl ```

[ricardobeat]

These are the first-level dependencies:

    chalk: log formatting
    dockerode: docker API
    download: file download?
    fs-extra: functions like mkdirp, emptyDir
    hasbin: check if bin exists in PATH
    js-yaml": yaml parser
    mustache: templating
    node-virtualbox: VB API
    progress: terminal progress bar
    simple-git: Git API
    sudo-prompt
    tar
    uuid

It looks reasonable at first sight. Of all these, fs-extra is the only one you can argue should be part of the standard lib.

A lot of the other dependencies are simply cruft, for example, `safe-buffer, `safer-buffer`, `buffer-alloc`, `buffer-alloc-unsafe` all patch the same issue and haven't been necessary since ~2016 / node 6.0. Same for `sort-keys`, object key sort order was baked into the ES2015 spec and has been the default behaviour since the beginning of the decade.

As mentioned by other commenters, this is a result of an extremely easy-to-use packaging system, coupled with a culture of sharing and reusability. The dark side of it is (justified) laziness and reinvention of the wheel - the core functionality of many of these modules can be written in one or two lines of code, but it is indeed faster and safer to just import something that exists and has been tested. The vast number of choices means it's hard to find standard solutions, and this also encourages developers to create their own 'improved' version of everything, in a self-reinforcing loop. The language itself has been in constant change, meaning new flavours of previously stable modules pop up to support new patterns (promises, generators, await, classes, etc etc). Then you get egos, marketing and corporate sponsorship added to the mix :)

[tylerl]

This is exactly why we don't allow our employees to use Node.js for company software development, even though in theory a fair chunk of the runtime is our own software that we maintain. But the npm ecosystem has a really awful signal-to-dogturd ratio, and developers appear to put very little effort into critically analyzing their dependencies.

The topic has come up, but I'm generally against it; sure, we could spend all the resources on doing all the filtering and analysis and change management that it would take to establish a sane package base that we could officially support for internal development work... But why? We do it for other languages, but Node has a lot of "my first programming language" bs going on in the ecosystem. I strongly suspect that allowing Node development would be a net negative for our company.

[untog]

I'm glad I don't work at your company. "developers appear to [...] I strongly suspect"... I hope the decision to exclude an entire programming language that's popular almost everywhere is based on serious study rather than appearances and suspicions.

[tylerl]

No, don't be silly; responsible companies don't pick and choose languages to exclude based on suspicions.

No... responsible companies exclude ALL programming languages except those that they can responsibly support -- you don't want the new guy writing some a critical piece of infrastructure in Haskell or Ada or Lisp or something because they feel it's morally superior, and then find out it can't integrate with some critical management system because it doesn't have the right bindings, or that it doesn't run on your upgraded production environment, or whatever. When you have tens of thousands of programmers, it's guaranteed to happen unless you proactively prevent it.

No, we have a handful of programming languages where we can guarantee that everything works, that the important infrastructure is accessible, that every library will remain supported until we replace it, that every vulnerability can be patched within a given timeframe, that code will run correctly on every machine, and so forth. Taking on a new language means dedicating a whole team of engineers to maintaining support for that language, indefinitely.

Some companies may be able to YOLO their way through decisions based on what what's popular at any given moment, but when you operate at any serious scale you have to be a bit more... responsible.

[montecarl]

This is what you get when you combine a really easy to use package management ecosystem and the standard library is really bad. You don't see this in Python for example, because its not so easy (its getting better though) to create packages and because its standard library is massive.

[krapp]

Javascript's standard library isn't bad, it's perfectly adequate for its intended use case - acting as a lightweight scripting language for the web, within a browser. Javascript as a system or backend language was a mistake, but there's no putting that genie back into the bottle.

[scrollaway]

You're confusing JS and NodeJS. NodeJS has more stdlib stuff; opening files, for example.

[cookiecaper]

Virtually everything is a dependency in JavaScript, whether you're using node.js or not. One need look no further than your citation of `open()` as the proof that the nodejs environment "has more stdlib stuff". If the operating environment can run in a context where file IO happens, a native file open function is about the most basic baseline there is. It's absolutely not a good look for "well, we have `open()`!" to be the poster child of the JS stdlib.

After `left-pad` and `flatmap-stream`, attempts to justify the state of affairs in the JS ecosystem are patent absurdities. They show starkly that the platform vendor needs to offer a reasonably-robust basic toolkit, and that cultures of "every function should be published as a library!" are a massive risk factor.

A good first-party standard library should be considered a security requirement for every application. Dependencies should be brought in with care and attention, not in a massive indiscriminate orgy of nested modules spraying every function into its independent own library and resulting in every node.js application requiring its own 500MB+ folder of libraries to even start.

Think it doesn't get worse? People are now using node.js to distribute "business cards". Arbitrary JavaScript execution on your local user account. Has science gone too far?! Someone at npmjs.com sure has by allowing this kind of thing. [0]

The technical world is crumbling. Who can fix it?

[0] https://blog.bitsrc.io/malicious-npm-development-kit-a02401e...

[untog]

> It's absolutely not a good look for "well, we have `open()`!" to be the poster child of the JS stdlib.

...which it isn't, given that the "fs" module has a wide variety of features much beyond an equivalent to open().

[ManuelKiessling]

Yesterday a friend of mine told me about how they got a Node.js application from a vendor that was about 3 MiB, and after running npm install, it was over 1 GiB.

I half-jokingly said that Node apps are the new ZIP bombs.

[stcredzero]

Back in the late 90's, I got into a USENET flame war, because the smallest, easily pruned ObjectStudio executable was an entire 4MB. Not small enough. Someone around that time got Squeak Smalltalk down to around 380KB. Not small enough. Back in the late 90's, you could still meet people who would insist that even your hyper-complex business app should be written entirely in C, because anything else was sinfully slow and wasteful. Any language with a VM was automatically too slow to be useful at all.

[xienze]

Isn’t that because every Node dependency stores its own dependencies within itself? So you could literally end up with multiple copies of the exact same version of the same library.

I’ve never understood why they didn’t go with the Maven approach: all dependencies stored in a central location, separated by version.

[dhritzkiv]

I believe the same version (within a specified semver range) are hoisted and stored in the root of ./node_modules. Differing versions are nested within the consuming packages and therefore duplicated.

This can be particularly bad when a popular package has a semver major change (even if, for example, support for an outdated version of node is dropped), many libraries will lag behind in updating to the latest major version and you will have many duplicated copies of a popular package.

[hn23]

good luck to your friend with repeatable installations :P

[news_hacker]

"If you wish to make apple pie from scratch, you must first create the universe"

- Carl Sagan

the npm ecosystem takes this quite literally, for better or worse.

[stcredzero]

I'll credit npm with this, truly, when there's a package for each one of the Peano postulates. (As an actual functional dependency.)

[JustSomeNobody]

If you need repeatable installations, wouldn't node be the wrong tool? I mean, you'd have to freeze everything yourself and then those libraries become _your_ problem. Ugh. That's a hell nobody wants.

[chrisbroadfoot]

To be pedantic, it's "wouldn't npm be the wrong tool" (it isn't, necessarily, I believe lockfiles provide you with reproducible builds)

Vendoring/copying them is another way to achieve this (and means you don't need to depend on npm or its lockfiles).

Regardless, those libraries are your problem whether you vendor/copy them or not.

Read more: https://research.swtch.com/deps

[quickthrower2]

You’d just have a local cache, you can pull updates from authors if needed.

[root_axis]

Is it really that strange? The transitive dependency graph for my iOS and Android apps are similarly complex.