[infogulch]

I'm annoyed by the false dichotomy that colors most discussions around package management that there are only two solutions to publishing software packages: 1. a carefully curated professionally maintained standard library, 2. the complete wild west where anything goes. It's not really "false" because this is the reality of how package managers are designed today, but it's false in the sense that it doesn't have to be this way.

You can see this tension in virtually every discussion, users resisting using packages that aren't published in the standard library for fear of attacks and poor quality, and maintainers that resist publishing in the standard library for fear of changing requirements and the appearance of better designs. Sure there are admissible entitlement / responsibility arguments against these respective positions, but that's mostly a distraction because both have a valid point.

The problem is that there's no space for intermediate solutions. We need packaging tools to aggregate and publish groups of packages that relate to a particular domain, and organizational tools to ensure quality and continuity of these package groups over time. This mitigates users' fears and reduces their cognitive load by curating the solution space, and it mitigates maintainers fears of ossification and backcompat hell by enabling them to create new package groups.

I'm saying there's an entire dimension of valid tradeoffs in this space, but the current design trend of package managers force us into one extreme or the other.

[jerf]

I'm unclear from reading your comment if you know this or not... but, what the Go team is describing is an intermediate solution. Not the exact one you describe, but it is intermediate. There is no particular requirement to get into the Go package ecosystem, no gatekeepers, it's all namespaced by the URLs you store your source code at, but between what the proxies do and the way the version requirements were specified, you are also not simply naked to every update someone somewhere pushes.

Please note I'm not claiming it's perfect or that you'll like every aspect of it, I'm just saying it is an intermediate solution between the two extremes.

[nonameiguess]

The original solution is the intermediate solution. You can build an entire useful userspace around nothing but libc and an ssl lib (take your pick between OpenSSL or GnuTLS usually). That is effectively what busybox is. Need to do some heavyweight math in Fortran? BLAS and LAPACK probably have everything you need. You can get really far with a C++ application using nothing but Boost.

For whatever reason, newer language ecosystems migrated away from that in the direction of increasingly smaller libraries until npm practically became a parody of it.

There is no reason you can't have lots and lots of useful functionality packed into a few large, well-maintained, well-packaged, well-vetted and trusted libraries, but you need trustworthy organizations willing to that maintaining and vetting. Historically, that seemed to largely be universities and research labs, where the funding and incentives are a lot different from the weekend warriors and solo devs that dominate open source landscapes today. Interestingly, I think library projects that still have large organizations behind them keep with the larger old-school ethos. Look at the world of ML and scientific computing. NumPy and SciPy are still huge libraries. Same with PyTorch and Tensorflow. QuantLib is an interesting example because it actually doesn't have a single large organization behind it. A bunch of Quants just got tired of doing the same things from scratch over and over and decided to aggregate their work for their common good. But it was 22 years ago, so maybe it was still just different back then and the trend toward small libraries hadn't kicked in yet.

[Tainnor]

I don't think your suggestion works for Web Development.

Imagine an application that talks to a Postgres DB, a Redis cache, several AWS services (e.g. S3, Lex) and also has to be able to parse excel documents. I've worked on applications like that. That's a whole lot of libraries you'll need to include. But I don't think you should include all of them in some "standard library", most people are not gonna need most of these dependencies.

I agree that left-pad is ridiculous (over in the JVM space, there is apache-commons, which also provides e.g. left-pad but of course it's not the only functionality that it provides), but using only 2-3 libraries isn't realistic either.

[throwaway894345]

This is kind of interesting. In the Linux world, you have package maintainers who (among other things) vet and vouch for the quality of the packages they maintain. I think there are similar things in the Docker ecosystem these days (since Docker really did/does seem to be the wild west).

It could be interesting if there was a similar concept for Go (and/or other ecosystems), except that instead of actually packaging the packages into artifacts (especially with the licensing headache that entails), it could be essentially a registry of verified package versions. So the "maintainers" in this sense are just validating the dependencies and maintaining a list of the approved dependencies (including their versions/checksums) and then automated tooling could be used by consumers ("consumer" here may or may not imply payment depending on whether this hypothetical venture is open or closed) to identify unverified dependencies in the consumer's project.

I'm sure someone has thought of this already--link me to relevant projects if you know about any.

[morelisp]

Seems like this would be equivalent to running your own module proxy / sumdb with a whitelist, which e.g. Athens can do.

[lawl]

> We need packaging tools to aggregate and publish groups of packages that relate to a particular domain, and organizational tools to ensure quality and continuity of these package groups over time

Whats stopping you? golang.org/x is kind of like that. Theres nobody stopping you from aggregating packages under foo.bar domain and build a reputation for high quality.

[infogulch]

Good question! The thing that the standard library has that I don't is version aggregation. The problem is not publishing under a single domain, the problem is publishing under a single version. Publishing a bunch of packages that I claim are high quality doesn't help users decide which versions to use, they still have to make this decision on a package-by-package basis. Note that I may be in the middle of a big redesign and some individually-complete packages use the new design but others don't, so you can't just say use the latest of each because they don't all work together yet. In this case I still want to publish each package individually for people who do want fine-grained control, but I wouldn't publish a new version of my "package group" with these versions because they're not integrated yet. My point is that aggregations of verified-interoperable packages is a separate problem domain and existing tools don't suffice to solve it.

[skybrian]

You could sort of do that using a Go module that points to all your other modules. Then anyone who depends on that will get the versions you specify (at a minimum).

But a problem is that they would also download all the modules you point at, whether they use them or not.

To fix that, the package system would need a "soft dependency" where, if a module exists, it must be at least the version indicated.

[infogulch]

Good idea with "soft dependencies". You'd also need to make the go.mod RequireSpec version argument [1] optional so it can be overridden by the module group.

    require example.com/my-module-group 1.2.3
    require example.com/some/module // selects minimum version specified by my-module-group

[1]: https://go.dev/ref/mod#go-mod-file-require

I also noticed Workspaces [2] which I hadn't seen before. They look interesting, but appear to exist for a different purpose. Maybe workspaces with a bunch of replace directives & some cli tooling could emulate a system like what is described here.

[2]: https://go.dev/ref/mod#workspaces

[throwaway894345]

I suspect licensing may stop you from doing this, but I'm not an expert. See my sibling comment for an alternative: https://news.ycombinator.com/item?id=30871181