Hacker News new | ask | show | jobs
by superasn 889 days ago
I've seen a lot of people criticise npm and their policies but I've never come across a solution. Npm has its flaws and while there are such abuses like everything package, is-odd, left-pad, etc there are also many useful packages like vue, sortable, etc without which development will be a huge pain.

So not asking rhetorically, if we had all the insight and knowledge we have now, how would you make it different?

18 comments

I know it's five hours later and this question has already spawned dozens of responses, but it's worth thinking and speaking clearly if we're trying to arrive at a solution for something. We can start by saying exactly what we're talking about—how do you make what different? Because you mention npm "and their policies" but then switch gears and talk about "is-odd", which is not a policy issue. It's rather something else entirely.

If you want answers, state clearly what _specific_ problem you're trying to solve. Whatever the solution to it might be, vague and fuzzy questions—while magnets for chatter since they can stand in for whatever someone wants to read out of them—are not the way to get there.

(You could say that this is needlessly tedious because everyone already knows what we're talking about, but that this isn't true is exactly my position. It's certain that something like half the people reading, thinking, and writing are have in mind one thing, while the other half are thinking of another—and the third half are thinking about something different from either of those. We're also programmers, so dealing with tedium and the constraints of having to be explicit should be second nature.)

> Whatever the solution to it might be, vague and fuzzy questions—while magnets for chatter since they can stand in for whatever someone wants to read out of them—are not the way to get there.

This is a great line. If HN had a quote of the month or something, this should be nominated for that.

You design a language for a purpose (which could be anything) you develop and mature it's features to better fit it's use case.

html, a crappy defective xml implementation refuses to grow up, js, while great for little html tweaks is not adopting any of the useful features found in popular npm packages. It was actively developed for 2 weeks. Ripping off it's head (nodejs) gave us a poor sailor jargon ~ but without the boats!

Therefore there is nothing wrong with npm, she is a fine ship. The harbor doesn't want to take it's much desired cargo, it must sail the 7 seas forever mon capitaine!

HTML came before XML. Also, how is HTML crappy when it is used by millions of web sites and is one of the most successful technologies in the past 35 years?

Does this mean it is perfect? No. Is it "crappy"? Nope.

Also, while JavaScript had a rushed development cycle, it has grown over the past 20-30 years and you can clearly write some great programs in it. Also, it has some very good features. My favorite is you can pass functions as variable arguments. It got this before a lot of other mainstream languages.

> Also, how is HTML crappy when it is used by millions of web sites and is one of the most successful technologies in the past 35 years?

That does not mean it's good.

Lead paint was widely used in the Europe and Americas for a very long time, doesn't mean it was good

The core of the problem is micro dependencies. It seems in the Javascript ecosystem, developers have no awareness of costs of complexity.

When you wonder whether to add a dependency, you should ask yourself: What are the upsides and downsides of adding this dependency. One downside is always that by adding a dependency, you add a potential security problem, a potential point of breakage, and more complexity.

There are situations where these are well justified. If your dependency is stable, from a trustworthy source, and if it is a functionality that you cannot quickly implement yourself. But if you include a dependency that is effectively one line of code, the question answers itself: The costs of adding a dependency is completely unreasonable. It your list of dependencies grows into the 100s, you're doing something wrong.

Devs working with core developers to create more 1st party packages would be a good start. I don't need 12 different implementations for sorting on Vue/React/[insert spa framework of the month]. I just need 1 really good sorting library. With it, we can move to less overall dependencies on random packages.
There are two massive reasons why js got here, with a million packages for tiny things and a culture of using them: browser cross-compatibility requiring complicated workarounds for easy-seeming tasks, and the introduction of promises + async/await to node.js after the standard library already used callbacks.

When you combine those together you end up with a situation where "normal" js code not from a library can't be trusted on the front end because it won't work for x% of your users, and offers a clumsy API on the backend that you'd prefer be wrapped in a helper. Developers learnt that they should reach for a library to e.g. deal with localstorage (because on Safari in private mode the built-in calls throw an error instead of recording your data and discarding it after the tab closes) or make a HTTP request (because node doesn't support fetch yet and you don't want to use the complicated streams and callbacks API from the standard lib) and they propelled that culture forward until everyone was doing it.

Modern JavaScript reminds me a lot of BASIC, Pascal and other 70s and 80s languages. Even C pre-ANSI.

We’ve been blessed in recent years that either languages are fully open source and come with a reference implementation, or a standards body governs the implementation detail. Sometimes even both.

Whereas JavaScript is really more a group of languages, each with their own implementation quirks.

ECMA was intended to bring some sanity to all of this. And it’s definitely better than it was in the JScript days of IE vs Netscape. But there isn’t anything (that I’m aware of) that defines What should be a part of the JavaScript standard library.

Wouldn’t it be great if there were a libc in the JS world. Something portable and standardised.

> Wouldn’t it be great if there were a libc in the JS world. Something portable and standardised.

I mean, it’s not necessarily “in the JS world”, but WASM is basically that.

WASM is explicitly not that. WASM itself has no APIs, it's just an execution envionment.

You may be thinking of WASI?

No it’s not. Problem right now is that every WASM files pulls in its own stdlib. Which is a waste once you use more than one.
> I don't need 12 different implementations for sorting on Vue/React/[insert spa framework of the month].

This feels like a bit of a strawman, since sorting is already in the standard library and there aren’t in fact popular sorting packages for each framework (that would in fact be ridiculous).

If you want to start a real debate though, bring up date/time pickers.

There are multiple date picker, time picker and datetime picker packages for each framework, and there are debates with good points on all sides about whether the browser-provided pickers are sufficient, or whether this is an area where a level of customization is needed and what that level is keeps changing as people discover new ways of designing date/time pickers and new use cases arise that require different tradeoffs. It’s both really frustrating but also kind of understandable.

There are still so many basic things that aren't in the JS stdlib, though. A good example is Map - if you need to use a tuple of two values as a key, you're SOL because there's no way to customize key comparisons. Hopefully we'll get https://tc39.es/proposal-record-tuple/ eventually, but meanwhile languages ranging from C++ to Java to Python have had some sensible way to do this for over 20 years now.
It’s not key comparison issue:

const idx = [1,2] const m = new Map m.set(idx,"hi!") console.log(m.get(idx)) // outputs "hi!"

console.log(m.get([1,2]) // outputs undefined

That last line has created a new array object, and Map is made to be fast so checks equality by reference. Ah, which is what you to be able to change. I guess you would want to pass a new map a comparator function, so that it does a deep equal. That would be faster than what you would have to do now:

const idx2 = String([1,2]) m.set(idx2, "yo") console.log(m.get(idx2)) // yo console.log(m.get(String([1,2])) // yo

That is precisely a key comparison issue. That is why I spoke about a tuple of two values; tuples by definition don't have a meaningful identity, so reference comparison is utterly meaningless for them.

Stringification is a very crude hack, and it doesn't even work in all cases - e.g. not if a part of your key still needs to be compared by reference. Say, it's a tuple of two object references, describing the association between those two objects.

Either way, the point is that this is really basic stuff that has been around in mainstream PLs for literally many decades now (you could do this in Smalltalk in 1980!).

There is a trivial way to have custom key comparisons: write a function that returns the key you want. You can implement equals() using some kind of serialization, or using a lookup table of references - whatever you want!

Of course, Records and Tuples would greatly simplify the process.

Writing a key comparison function is not a problem. The problem is that Map does not have any way to use such a thing; it always compares using a predefined equality algorithm that is by-reference for all aggregate data types.
I didn't say that you could pass a key comparison function directly into Map.

What I meant was that it's possible to emulate a custom key comparison predicate.

You just have to have a function that returns value for each input that behaves the way you want under strict equality comparison.

Implementing an `equals` function that returns a boolean is more convenient, sure.

Serializing (for plain objects with only JSON-serializable values that would be JSON.stringify) to strings or other primitives would of course be possible with object keys, too. But that's probably what you want for "record"-like objects, right?

And if you want better performance or compare non-primitive values, you'd have to do something more complex, that's what I meant by the lookup table.

But I imagine if you deep-compare large Record objects a lot, the performance wouldn't be any better, because the engine still has to do a deep comparison.

If I am not mistaken, Records/Tuples are in fact strictly limited to this case:

https://github.com/tc39/proposal-record-tuple#jsonstringify

So basically there is no difference to having a function serialize() that just stringifies your plain object, maybe with a caching layer in between (WeakMap?)

OK, thinking about it, the proposal really would help to avoid footgun code where performance optimizations are lacking, and too many deep comparisons or too many serializations are performed.

> since sorting is already in the standard library

Right. For example, to sort a list of numbers:

  [5, 14, 1, 2].sort()
Works great! No, wait, that's obviously wrong. Uh,

  [5, 14, 1, 2].sort((a, b) => a - b)
> There are multiple date picker, time picker and datetime picker packages for each framework, and there are debates with good points on all sides about whether the browser-provided pickers are sufficient,

Safari (iOS and MacOS) still doesn't have full support for the date time picker, which is why there are so many alternatives.

That’s a really good way to stagnate imho. I’d rather have 10 sorting libraries that each specialize or make different trade-offs than one library that tries to do everything.

That said, you can still have a core set of “blessed” packages that serve the common needs.

I don't see how creating a definitive sorting library is stagnation compared to having 10 mediocre libraries that are all missing some sort of critical functionality.
Your argument seems to be "just write good code instead of bad code". My argument is "the best way for good code to exist is to enable and support multiple options". Because if you have only one option and it's bad then you're screwed with no recourse. C++ and Python have, imho, many horrible API designs and we're stuck with them forever. This is stagnation.

Rust has a good standard library and also a large community of libraries. Sometimes those community libraries get promoted to std because they're strictly better. Sometimes the std version of hashmap is slow because std insists on using a crytographically secure hash when 99.99% of use cases would be better served with a less secure but faster hashing algorithm.

Like many things in life the ideal scenario is a benevolent dictator that only makes good choices. In practice the best way to get something good is to allow for multiple choices.

<insert parable of pottery class graded on quality vs quantity>

The problem with this argument is that JS also has many horrible API designs. That seem to be replaced by equally horrible API designs, just with a faster churn.

Meanwhile, the horrible C++ and Python API designs at least offer the needed functionality, even if the code looks ugly.

Forgive me for nitpicking your Rust example but you can define your own hashmap that inherits from the standard hashmap, and give it a different hash function. I have done it.
Right. I'd be very surprised if anyone looks at languages with strong standard libs and says "I wish they had the kind of sorting I can pull in from npm"
Because who is going to bother working on any packages if they risk rejection in the end? Have fun with your ______ package because it's never going to improve.
There’s plenty of languages with vast standard libraries and which also have 3rd party libraries that offer the same feature as something in stdlib but more enhanced against a specific metric.

You see it in Go, Python, .NET, etc.

Also Java and Kotlin.
A well designed JS standard library that also includes a set of protocols (interfaces) would make such a huge difference in QoL. It would also likely be the biggest contributor to reducing bundle sizes. The protocols (iterable, async iterable etc) will ensure that the rest of the ecosystem can also innovate and participate at a similar level of ergonomics by implementing them
> A well designed JS standard library

While I agree here, you also have to remember that additions to the JavaScript standard also increase the amount of time / effort for new browsers to enter the space.

The JavaScript standard (the web APIs, mainly) are already very complex, with Web Workers, Push Notifications, Media Streams, etc. that additions to it should be made cautiously -- once an API is implemented, it's there forever, so the bar for quality is much greater than that of some NPM library.

A JS standard library would be a drop in the bucket compared to the size and complexity of the DOM libraries and implementing a usably performant JS engine.

Yes it should be done carefully. There are also plenty of examples of how this can be done well, done by experienced engineers. For example, the Dart starndard library (https://dart.dev/libraries - core [1], collection [2] and async [3] in particular) is a very good model that would fit JS fairly well too (with some tweaks and removals)

[1]: https://api.dart.dev/stable/3.2.4/dart-core/dart-core-librar...

[2]: https://api.dart.dev/stable/3.2.4/dart-collection/dart-colle...

[3]: https://api.dart.dev/stable/3.2.4/dart-async/dart-async-libr...

> A JS standard library would be a drop in the bucket compared to the size and complexity of the DOM libraries and implementing a usably performant JS engine.

It's still a nonzero amount of complexity. I see a lot of "v8 is really hard to compete with" comments on here so this feels very pertinent to mention. You can't have it both ways.

> Yes it should be done carefully. There are also plenty of examples of how this can be done well, done by experienced engineers. For example, the Dart starndard library (https://dart.dev/libraries - core [1], collection [2] and async [3] in particular) is a very good model that would fit JS fairly well too (with some tweaks and removals) > > [1]: https://api.dart.dev/stable/3.2.4/dart-core/dart-core-librar...

This one, at least, looks somewhat inspired by JavaScript.

There's a difference between features that need to be implemented as part of the engine such as Web Workers and those that can be implemented as a library, such as sorting; the latter can be shared between implementations much easier.
If that standard library would be written in JS, a new browser (or rather a new JS engine being a part of the browser) could just use some existing implementation (a reference implementation maybe?), no need to reinvent the wheel in every part of the browser.
> If that standard library would be written in JS, a new browser (or rather a new JS engine being a part of the browser) could just use some existing implementation

That sounds great, but I'm doubtful of the simplicity behind this approach.

If my understanding is correct, v8 has transitioned to C++[0] and Torque[1] code to implement the standard library, as opposed to running hard-coded JavaScript on setting up a new context.

I suspect this decision was made as a performance optimization, as there would obviously be a non-zero cost to parsing arbitrary JavaScript. Therefore, I doubt a JavaScript-based standard library would be an acceptable solution here.

[0]: https://github.com/v8/v8/tree/main/src/runtime [1]: https://v8.dev/docs/torque-builtins -- As I understand it, Torque compiles to C++ at compile-time, which is then linked and compiled into the rest of v8[2]. [2]: https://github.com/v8/v8/blob/main/tools/torque/format-torqu...

Hasn't everyone pretty much given up on making a new (standards compliant) browsers after Microsoft gave up?

(Or are they still trying to make Servo viable?)

> Hasn't everyone pretty much given up on making a new (standards compliant) browsers after Microsoft gave up?

There's plenty of competition, even if the current projects are in a beta (or even alpha) state. Consider the LadyBird browser developed by SerenityOS, or Servo.

I don't know, I think "batteries included" standard libraries got a bad reputation because Python's standard library is so full of crap, so lots of people thought the whole idea was fundamentally bad. But I think the correct conclusion is just that Python's standard library is bad.

Go has a big standard library too and it's mostly very well designed, useful and avoids fragmentation.

I think a similar thing happened with compiler warnings and C/C++. The language is error prone so people want warnings but a lot of the warnings don't have good solutions (e.g. signedness mismatches) so people tend to ignore them. Also they aren't easy to control, e.g. from third party dependencies.

So some people got the idea that warnings are fundamentally wrong and e.g. Go doesn't have them. But my experience of Rust warnings is that they are totally fine if done right.

Don't allow un-publishing package versions. If they are literally malware, they can be manually removed by npm admins. If a court orders a takedown due to copyright, that's also something npm admins can handle. If you want to be able to un-publish, then just publish on your own server (or github etc).

If analyzing the dependencies for showing in the NPM web UI, while analyzing, as you exceed 40 direct or transitive dependencies, abort and highlight this package in red, for having excessive dependencies.

If installing locally, you get what you get, don't install random or crazy packages, stick to well known high-quality minimal-dependencies packages. nodejs does include file reading and writing, http server, http client, json ... that will take you pretty far. Master the basics before getting too fancy. And remember, you don't need some company's client package just to make some http requests to their API.

I’ve started thinking package management has too much trust now. Ideally, but probably unpractically, projects should check in their packages like they used to under /lib or /third_party, and be much more suspicious of new package dependencies.

Basically, you would need to start accepting that you are responsible for any dependencies you choose to include. Any upstream changes you would need to evaluate and bring in or patch yourself.

Definitely an impossible task given how broad and deep modern package dependencies are, but at least you’d start feeling the insanity of having all if them in the first place :P.

If NPM made some tweaks this might become trivial. Keep a node_modules/packagefiles with all the .zips that you commit to your source control. The expanded files can be kept out as they are now recoverable just using zip!
> Keep a node_modules/packagefiles with all the .zips that you commit to your source control. The expanded files can be kept out as they are now recoverable just using zip!

What problem does this solve?

Wouldn’t the opposite be better? I’m not sure you could take advantage of the vast majority of files in the zip files being unchanged if you kept compressed archives.
Not sure what you mean but typically you don’t need to track changes of libraries to that level. At least not in the context of a repo using those libraries. I am thinking of treating them like binary .dlls.
Source control is really good at compressing text files as they evolve over time, but isn’t optimized to handle binary assets. Since a single-line patch changes the entire zip archive, you’d risk growing the size of the repository based on the number of patches.
Yarn does that in some modes.
IIRC, the Maven crowd was criticizing npm's decisions from the get go because they chose to ignore many of the problems the Java community already solved a decade before.
So could you list those problems?
One issue is npm will allow arbitrary code to execute as part of an install script for a package, which allows a class of attacks that aren't possible in the maven world.
Namespaces, for one.
Namespaces in Maven seem like they're clunky. The pseudo-DNS thing where the first section is an actual domain but the second is whatever you want is quite janky, as is not matching the namespace to the package. Plus domains are transferable themselves and it seems like a bad idea to use them as identifiers.

Not to say that npm shouldn't have had namespaces by default, but I think there's good reason not to blindly do everything the way the Java community did.

> Not to say that npm shouldn't have had namespaces by default, but I think there's good reason not to blindly do everything the way the Java community did.

I'm not saying that they should have blindly copied Java, but they should have had something.

The entire reason this is a big deal is that people don't know what their dependencies are. The left-pad incident wasn't a big deal because it was pulled, it was a big deal because no one could easily fix their builds and didn't even know they were depending on it, because it was a dependency of a dependency of a dependency.

While it's ridiculous to expect that people will audit every single dependency and sub-dependency, it's not ridiculous to expect tooling to do the same.

Packages should be given an overall quality rating (and honestly it might be great for an ecosystem as large, diverse, and welcoming-to-beginners as JS/TS), part of the score comes from the number of different dependencies/sub-dependencies -- a social package score if you will. If a package causes the dependency graph to explode, give a warning before installing it.

Then, if you're NPM, you don't need all of these convoluted and exploitable policies around un-publishing.

> While it's ridiculous to expect that people will audit every single dependency and sub-dependency

It's not ridiculous at all. Professional programmers should answer for the dependencies they bring into their projects.

This is why I prefer vendoring dependencies. I have to actually code review them.
Whether you're storing your own copy of a given dependency and whether you've done code review for it are orthogonal concepts. (You can check it in and perform the same amount of review that people do when deferring to `npm install` for late fetching, i.e. none.)

Conflating these two not-unrelated-but-still-distinct concepts is a big contributor to why the current state of the art is so fraught.

I'd be called insane if I suggested it. I work with dotnet and I'd rather not add all the code in newtonsoft json and manually review each line. I mean where does it stop? Why not have everyone in the world code review asp.net and dot net libraries for every single website project at that rate?
Rust’s Cargo vet offers an answer to that question.

You can import a list of audits from trusted auditors, which should cover all popular packages. Now you have to audit dependencies that aren’t well-known in the community, which really is the set of dependencies that you should take an extra look at. The big popular JSON libraries can be audited by either Microsoft or some of the other large projects that are using them.

You’d explicitly share your trust list in your audit file, and anything (updates or new packages) that isn’t trusted by you or one of your listed auditors is flagged for auditing.

https://mozilla.github.io/cargo-vet/index.html

Personally I would like it and the ecosystem to just cease to exist overnight. Nothing on earth has caused so much pain, misery, suffering and agony, apart from possibly PHP.

Our devops guys scream from the seething pain whenever the have to debug some pile of shit that decides it won't build unless all the runes are aligned precisely and all the RAM in the universe is available on the build runners. And pushing this to the developers results in importing more packages thus adding to the burning tyre fire.

And after several hours of builds and 9000 layers of packages you wake up one morning and in that 50 meg chunk of javascript that is excreted from the process, someone managed to inject a "Slava Ukraini!" banner into your web app.

I'm sorry, but there is no chance that the automake/autoconf suite of tools has caused less pain than `npm install`.
We have almost buried those.
I mean... just don't install everything then?

Over-reliance on third party dependencies is a choice. One could argue that it's unreasonable not to do it if you want to stay competitive but good luck changing human nature then. If there are shortcuts, they will be taken.

While this is true, when the shipped standard library by NodeJS lacks so many VERY BASIC features that every other language has, OF COURSE developers choosing/being instructed to use JS so that frontend and backend languages are synced are going to reach for packages to solve whatever functionality should already exist in code like: "Jim".leftPad(4)
"Jim".padStart(4) has existed in JS (and Node) for 6 years now.

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...

The one thing clear in JavaScript is that if some developers think there is a better way, they will develop it and use it. That hasn't happened with NPM because its about as useful as it conceivably could be. The criticisms really amount to nothing, and tend to come from developers who don't even write JavaScript.
> how would you make it different?

Make the cost of reusing software non-zero again.

It doesn't have to be as painful as C++ without package managers, but should make every developer spend about 5~10 minutes labor work for adding each direct dependency, or one minute for each new dependency in the dependency closure.

When you use 'go get' to add a package to your Go project, it actually fetches the code through a Google proxy which saves a snapshot of the commit in question. Even if the original source goes away, they should have a copy of every version of the library ever fetched via their tool, and devs can continue to build existing stuff.

(If you don't want Google to see what packages you're fetching, you can also turn this off with an environment variable.)

I never used Node and went right to Deno, partly because NPM and packages sounded like a mess. So far it has been a good experience.
I don't know that there's a solution because the fundamental cause of the problem is that Javascript has a huge dev base and everybody wants to have at least one active NPM package they maintain for their resume. Nobody ever asked me as a Perl programmer what CPAN packages I've created because very few Perl programmers made them, but hiring managers will look at Javascript devs' NPM footprint.
i don’t know how you fix this for js, but in general i think well designed and robust standard libraries are a great place to start.

the community shouldn’t need to write a bunch of tiny utility packages to do common things.

in other words, make it easier to avoid the deeply nested dependency mess that js encourages.

This problem is a symptom of "move fast, break things" mentality that pervades the JS (and, more broadly, the web) ecosystem. The result is an ecosystem that is specifically optimized for moving fast and breaking things - which is a lot easier when the stable core is tiny.
The solution is a proper deprecation mechanism with a grace period for migration. Restricting removal or allowing instant removal are the extremes that cause trouble.
how would you make it different?

Have smarter users. If your package breaks because it depends on trivial code which got deleted, you shouldn't have depended on that in the first place.

Preventing people from deleting their code -- always, or even just sometimes -- was never the right solution.

I know "have smarter users" sounds like a joke but a lot of the problem really is cultural. In most languages you would write a two line leftpad function, in the js world everyone will tell you you're doing it wrong and should use a library.
Honestly, how go does package management is pretty good.
It's personal tastes perhaps, but I don't find the appeal of packages management in Golang. I find PIP, NPM, RubyGems, Nuget, Cargo,… easier to work with. The go.mod syntax is what it is, and doing updates or fixing conflicts isn't easy.

Not having a registry is neat, but I'm also unsure of what is going to happen over time as dependencies may be moved or removed. You can see that with old Maven pom.xml where some dependencies do not resolve anymore.

> I'm also unsure of what is going to happen over time as dependencies may be moved or removed

That's what the Go module proxy is for. The authors can move or remove their repositories as much as they want, I as a dependent am not bothered by it. They would have to go through an official vetted process to get it removed from the proxy.

Oh I didn’t know that the proxy.golang.org was a thing. That’s good to know.
This is why you should vendor your dependencies. They are part of your codebase at that point.
I have better things to do though. This feels unnecessary.
“go mod vendor”.

That’s it. That’s all you need to do.

Oh like this. I’m not sure I would enjoy overloading my git repository and merge requests with the dependencies. I was thinking about having a proxy or forking all the dependencies.
Soft deletes. You can delete a package and it stops being advertised but a shadow copy of referenced versions are kept for anything that depends on it. NPM spews warnings when this happens.

Once the referencing packages are updated are deleted or modified the shadow versions can be dropped.