[Athas]

Vgo has certainly been a disaster from a communication and community point of view. From an outsider perspective, it looks like a case of maintainer arrogance, and it is hard to believe that Go really has needs that are significantly different from those of e.g. Rust or Java.

However, I am also in the unusual position that I had to design and implement a package manager for my own little language recently. My original hunch was to just create a Cargo clone, but after reading Russ Cox's writings, I ended up cloning vgo (with a few simplifications). The implementation was wonderfully simple - everything just fit together. The algorithms are trivial. Operationally, it is easy to understand what the package manager is doing (even more so than for vgo, as vgo has do deal with various Go idiosyncracies and backwards compatibility).

This is in stark contrast to my experience with other package managers, which are temperamental beasts at the best of times. When they work, everything's groovy, but their error modes are easily incomprehensible. I think vgo's approach of restricting expressivity and streamlining processes is the right one. But since nobody has used such a package manager before, it remains to be seen whether it works in practice.

For my own experiment, I do have one data point: I showed people the (rather brief) documentation for my vgo-inspired package manager, and they all felt it was very simple to follow and easy understand what it did, and how.

[zeeboo]

For what it's worth, I analyzed as many Gopkg.{lock,toml} files I could find in the wild, and found that vgo's algorithms would service all of them. I've done similar analysis for a smaller number of Rust projects using Cargo and had the same findings. I've found it hard to find projects in the wild that actually do any non-trivial version selection. I think this is some good evidence that it will work out in practice.

I think it's less that Go has significantly different needs, but it's more that people overestimate what their actual needs are.

https://github.com/zeebo/dep-analysis

[Athas]

> I think it's less that Go has significantly different needs, but it's more that people overestimate what their actual needs are.

I think you are right. And I doubt it hurts that SAT solving is a fun problem!

My main package management experience has been with Haskell, which has used the cabal tool for many years. Cabal was a traditional solver-based tool (with the added pain of a global mutable package database, although that is going away), and it frequently broke down in confusing ways. Cabal hell was a widely used term. A few years ago, another tool arrived on the scene, Stack, which used the same package format and such as cabal, but snapshotted the central package list (Hackage) by gathering subsets of packages and versions that were guaranteed to work together (or at least do not have conflicting bounds). It works well[0], and although it does in principle result in a major loss in flexibility, it's rarely something I miss. Importantly, the improvement in reliability was nothing short of astounding. That certainly helped convince me that flexibility may not be a needed feature for a (language) package manager.

[0]: There are all sorts of socio-political stack/cabal conflicts in the Haskell community now, but I'm not sure they are founded in technical issues.

[pcwalton]

The problem with minimal version selection isn't that it can't support nontrivial version selection but rather that it doesn't automatically select the newest versions of packages.

[kaeshiwaza]

It's a feature not a problem. It select the only know working solution. You're still free to update to newest versions.

[pcwalton]

I've never seen Cargo errors be incomprehensible. Solving versions just isn't a problem in practice. Minimal version selection optimizes for convenience of the package manager implementer at the expense of the users. To me, that's the wrong trade.

[pas]

Could you contrast Cargo and vgo for those who only know the former?

[Athas]

There are lots of small differences, but I would say there are two major:

* Semantic Import Versioning. A package is identified by some name. I think vgo calls it an "import path", but I have also seen "package path" used. Every version of that package must remain compatible with previous versions. You may not break compatibility (i.e. increment the major semver number) without also renaming the package. There is some syntactic sugar that makes it clear that a version 2 is closely related to the original version 1, but from the point of view of vgo, the two versions are completely distinct packages. This also means a program may depend on several major versions of the same package (which Russ claims is necessary when doing gradual migrations of large programs).

* Minimal Version Selection. In contrast to pretty much every other package manager, vgo tries to install the oldest version of a packages that still satisfies any lower bounds specified anywhere in the dependency tree. Upper bounds are not supported. This both makes the solving algorithm trivial (which probably doesn't matter much), but also gives you reproducible builds without using a lockfile (which is nice), and it gives you a very simple operational model for what the package manager is doing (which I think is crucial).

All package managers suck and all package managers break, but I think when vgo breaks, it will be more obvious what is wrong. Time will tell whether it works in practice!

(And I'm not even a Go programmer; I just like the thoughtfulness that goes into the tooling.)

[infogulch]

Adding to this:

> vgo tries to install the oldest version of a packages that still satisfies any lower bounds specified anywhere in the dependency tree

Another benefit of this that may be a little less obvious (it wasn't obvious to me) is that this means every package-version selected by MVS in the entire dependency tree is explicitly specified somewhere in the dependency tree. This means that every selected package-version was specifically tested with at least one other package (assuming, reasonably, that libraries are tested with the versions that they specify). You don't get this with SAT, in fact it's possible for SAT to select a set of packages where no two package-versions were ever tested with each other anywhere else before.

I feel that this property, combined with upgrades requiring affirmative action by the root package maintainer (by editing go.mod or having a tool do it), will make vgo-managed packages much more reliable over time.

[lilyball]

> but also gives you reproducible builds without using a lockfile (which is nice)

I find the claims that vgo removes the lockfile disingenuous. Yes it's technically correct in that there's no lockfile, but it's incorrect in that you've basically turned the file that declares dependencies into the equivalent of a lockfile.

With something like cargo, I can `cargo update` and it will fetch new versions of my dependencies and update my lockfile. Net result, I have one file to commit changes to (the lockfile), with zero manual edits.

With vgo, updating a dependency requires manually editing the dependency list to specify the new version. Net result, I have one file to commit changes to (the dependency list), but it required manual editing.

In both scenarios, the one file that I have to commit changes to specifies the version of the package that will be used. Though that's not actually strictly true with vgo; there it specifies the minimum version that will be used, but that's not necessarily the actual version if another dependency requires a later version. Not a big deal, but it does mean there's no single file I can inspect to find out what the resolved package version is.

[skybrian]

If you want the latest version using versioned go, you have to run a different command. "go get -u" will upgrade to the newest version and also update the dependency list. [1]

So that difference isn't actually all that different.

[1] https://tip.golang.org/cmd/go/#hdr-Module_aware_go_get

[EdiX]

go get will update a dependency and edit go.mod for you automatically. And using go mod -fix will fix misleading versions specified in go.mod (i.e. if you require version X but are getting version Y because of a transitive dependency the require will be updated to say Y).

[kenhwang]

I'm still mind-boggled by the crusade against the lockfile. It's a completely unambiguous source for everything that costs essentially nothing, why is there so much time and energy devoted to getting rid of it? Especially with all the drawbacks minimal version selection comes with; exact version selection with the lockfile seems better in just about every case.

[Athas]

I don't think the lockfile is a big problem (and I don't have enough experience with lockfile-based systems), but if you already have some other file that can serve the same purpose, isn't that nicer than having two? It also means you don't have to be confused about whether lockfiles should be committed to version control (apparently the answer is "not always", which is counter-intuitive to me).

[bostonvaulter2]

When are you not supposed to commit a lockfile? I think the norm is to commit it.

[eridius]

You're supposed to commit it for applications, but omit it for libraries. Libraries declare the versions they're compatible with but they don't lock to specific releases.

[e12e]

I had some experience with plone (and zope) - big, early python projects that formed a lot of the basis for package management and infrastructure for python.

The "known good set" of packages as used by buildout for any given plone version were not really easy to work with.

See eg:

https://docs.plone.org/4/en/manage/troubleshooting/buildout....

Which points to things like: http://dist.plone.org/release/3.3.5/versions.cfg

Which aren't that much worse/better than a gemfile.lock for a rails project, I guess.

Now, I guess the better answer is to have smaller projects, with different interfaces (eg: don't give all modulles/parts access to your zodb object database, have a ome json/rest interfaces etc).

But apparently there are still many "big systems" being built.

[jessaustin]

Minimal Version Selection.

That's really interesting. Obviously this would lead to fewer regressions, but it seems opposed to the "security problems fixed for free" scenario you hear about with other package dependency schemes. Were we all just imagining that scenario? Is there another way we should be handling vulnerabilities, i.e. by "repudiating" old versions rather than simply releasing new ones? Then you could have something like "Minimal Unrepudiated Version Selection".

[Athas]

I think reproducible builds are more important than having security problems fixed behind your back. It's not like vgo makes it hard to upgrade all dependencies to their most recent version - it's done automatically with a single command, if you want it.

[necubi]

Other package managers (cargo, gemfile, etc.) also give reproducible builds by using a lockfile. Upgrading dependencies to their maximum version is an explicit action (cargo update). It's not clear to me why the go team things lockfiles are a huge issue.

[kjksf]

Lock files also remove the "silent security fix" that was touted as the benefit of other systems.

You can't have it both ways.

You either get reproducible builds (by default with vgo, by adding lock files in other systems) or you get silent upgrades that potentially fix security issues (but also potentially introduce security issues).

You also mis-characterize the position of vgo creator. He doesn't think that lock files are a huge issue per se.

The difference is in default behavior of the system: vgo by default picks predictable, consistent version of dependencies. That version doesn't change if dependencies release new versions.

In fact, it's not just the default behavior but the only behavior.

Other systems allow specifying complex rules for which version of dependency to pick and they all allow for a scenario where you run the same algorithm over the same rules but pick different version because in the meantime some dependency has released a new version.

It's such a big problem that all those system end up introducing lock files, which is a tacit admission that what vgo does by default is a good idea.

vgo doesn't need lock files to get the benefit of lock files, which is a nice cherry on top but the real advancement is in changing the default behavior of how resolving versions of dependencies work.

[jessaustin]

What you're saying makes perfect sense. This is what makes HN great. Incidentally, there's nothing about e.g. npm that would prevent this scheme: just set literal version numbers instead of using operators. The two systems definitely have different "best practices".

[pcwalton]

> It's not like vgo makes it hard to upgrade all dependencies to their most recent version - it's done automatically with a single command, if you want it.

And how does it guarantee that this actually results in anything that works, if upper bounds are not supported?

[kaeshiwaza]

Upper bound are major version, everything between should be api compatible (it's in the specs of go modules).

[Merovius]

> Minimal Version Selection. In contrast to pretty much every other package manager, vgo tries to install the oldest version of a packages that still satisfies any lower bounds specified anywhere in the dependency tree.

This is somewhat misleading. Most package managers use a human-edited manifest containing constraints and a machine-edited lockfile, containing the chosen versions. You periodically (or after changing dependencies to require newer versions) run a tool that fetches the newest versions satisfying your constraints and writes out a lock-file, which you then commit and use at install-time to get reproducibility.

Go modules use a single file that serves as both and is both human- and machine-edited. You periodically (or after changing dependencies to require newer versions) run a tool that fetches the newest versions satisfying your constraints and updates the manifest with them. The manifest is then (also) used at install-time with minimum version selection to get reproducibility.

What that means is that in practice, Go modules never install older versions than the equivalent in other package managers would do. The maintainer specifies constraints and uses a tool to decide what version will actually used to at build-time. In both cases, that tool will choose the newest versions available.

What is correct (and contentious) is that Go modules have no concept of upper bounds for dependencies. So the complaint is, that as a library author you can not prevent a newer version from being chosen by one of your reverse dependencies. It remains to be seen how much of a problem that will be in practice.

So, if anything, the problem is that as opposed to other package managers, Go modules sometimes choose too new versions, from the perspective of some people. It never chooses too old versions.

[nathancahill]

> vgo tries to install the oldest version of a packages that still satisfies any lower bounds

This alone would make npm 100x less painful to use.

[jessaustin]

You don't have to use operators in package.json, so this pain relief is available to you right now. Just let the version remain precisely as "npm install" wrote it, with no tildes or wildcards or anything.

Admittedly, the "update all packages to latest compatible version" flag described above sounds very nice. There is approximately zero chance the npm developers would ever accept such a PR, but there would be nothing wrong with a utility that accomplished the same thing.

[reggieband]

That may work for my dependencies but not the dependencies of my dependencies. In other words, for that to work it isn't enough for me to change - every npm library maintainer must change.

[ithkuil]

Yep, this is the problem with "features", when they exist somebody else will use them and you'll eventually have a dependency that does.

[jscissr]

Maybe deno will implement something like vgo. (If you don't yet know about deno: https://youtu.be/M3BM9TB-8yA )

[faitswulff]

That's just the thing, right? It seems to be a clearly superior technical solution, but it has thoroughly failed from the perspective of human relationships.