[munificent]

Let's say my_app uses package foo which uses package bar.

It turns out there is a security bug in bar. The bar maintainers release a patch version that fixes it.

In most package managers, users of my_app can and will get that fix with no work on the part of the author of foo. I'm not very familiar with Go's approach but I thought that unless foo's author puts out a version of foo that bumps its minimum version dependency on bar, my_app won't get the fix. Version solving will continue to select the old buggy version of bar because that's the minimum version the current version of foo permits.

[coder543]

> I thought that unless foo's author puts out a version of foo that bumps its minimum version dependency on bar, my_app won't get the fix. Version solving will continue to select the old buggy version of bar because that's the minimum version the current version of foo permits.

That is incorrect. The application's go.mod defines all dependencies, even indirect ones. Raise the version there, and you raise it for all dependencies. You cannot have one more than one minor version of a dependency in the dependency graph.

[munificent]

That still implies that I need to know to update the version constraint of what may be a very deep transitive dependency, doesn't it?

[coder543]

The version constraint is always listed in your top level go.mod file, so you know the dependency exists, no digging into the dependency tree required at all, and it’s not hidden in some lock file no one ever looks at. Plus, there are plenty of tools that help you with this problem, including the language server helping you directly in your editor and Dependabot on GitHub.

I’m not aware of any languages that send you an email when your dependencies are out of date, so yes, you need to check them. Dependabot can do this for you and open a PR automatically, which will result in an email, so this is one way for people to stay on top of this stuff even for projects they deploy but don’t work on every single week.

If you’re suggesting that indirect dependencies should automatically update themselves, then you are quite literally saying those code authors should have a shell into your production environments that you have no control over, compromising all your systems with a single package update that no one but the malicious author got to review. It is possible with tools like Dependabot to be notified proactively when updates are required so you can review and apply those, but it is not possible to go back in time and un-apply a malicious update that went straight to prod.

Repeatedly assuming that the Go core team never thought through the design of Go Modules and how it relates to security updates is such a strange choice. Go is a very widely used language with tons of great tooling.

[munificent]

I'm sorry but I'm not super familiar with the workflow for working with dependencies in Go, I've only read about it. You say:

> Raise the version there.

Am I to understand that it's common to hand-edit the version constraint on a transitive dependency in your go.mod file?

But that transitive dependency was first added there by the Go tool itself, right?

How does a user easily keep track of what bits of data in the go.mod file are hand-maintained and need to be preserved and which things were filled in implicitly by the tool traversing dependency graphs?

> Repeatedly assuming that the Go core team never thought through the design of Go Modules

I'm certainly not assuming that. But I'm also not assuming they found a perfect solution to package management that all other package management tools failed to find. What's more likely is that they chose a different set of trade-offs, which is what this thread is exploring.

[Vendan]

> Am I to understand that it's common to hand-edit the version constraint on a transitive dependency in your go.mod file?

No, run `go get <package with vuln>@<version that fixes vuln>` and Go will do it for you.

[na85]

>No, run `go get <package with vuln>@<version that fixes vuln>` and Go will do it for you.

The point GP was making is that it's not a given you'll know that there's a security vuln in a sub-sub-sub-dependency of your app. Is it reasonable to expect developers to manually keep tabs on what could be dozens of libraries that may or may not intersect with the dependencies of any other apps you have on the go?

Maybe for Google scale where you can "just throw more engineers at the problem".

[coder543]

> Am I to understand that it's common to hand-edit the version constraint on a transitive dependency in your go.mod file?

"Common" is probably not accurate, but what's wrong with hand editing it? If you mess up, your project won't compile until you fix it. You can update the versions a dozen different ways: editing by hand, `go get -u <dependency>` to update a specific dependency, `go get -u ./...` to update all dependencies, using your editor to view the go.mod file and select dependencies to update with the language server integration or by telling the language server to update all dependencies, by using Dependabot, or however else you like to do it. The options are all there for whatever way you're most comfortable with.

> But that transitive dependency was first added there by the Go tool itself, right?

So what? It's still your dependency now, and you are equally as responsible for watching after it as you are for any direct dependency. Any package management system that hides transitive dependencies is encouraging the user to ignore a significant fraction of the code that makes up their application. Every dependency is important.

> How does a user easily keep track of what bits of data in the go.mod file are hand-maintained and need to be preserved and which things were filled in implicitly by the tool traversing dependency graphs?

You already know[0] the answer to the most of that question, so I don't know why you're asking that part again. As a practical example, here is Caddy's go.mod file.[1] You can see the two distinct "require" blocks and the machine-generated comments that inform the reader that the second block is full of indirect dependencies. No one looking at this should be confused about which is which.

But the other part of that question doesn't really make sense. If you don't "preserve" the dependency, your code won't compile because there will be an unsatisfied dependency, regardless of whether you wrote the dependency there by hand or not, and regardless of whether it is a direct or indirect dependency. If you try to compile a program that depends on something not listed in the `go.mod` file, it won't compile. You can issue `go mod tidy` at any time to have Go edit your go.mod file for you to satisfy all constraints, so if you delete a line for whatever reason, `go mod tidy` can add it back, and Go doesn't update dependencies unless you ask it to, so `go mod tidy` won't muck around with other stuff in the process. There are very few ways the user can shoot themselves in the foot here.

Regardless, it is probably uncommon for people to manually edit anything in the go.mod file when there are so many tools that will handle it for you. The typical pattern for adding a dependency that I've observed is to add the import for the dependency from where you're trying to use it, and then tell your editor to update the go.mod file to include that dependency for you. The only time someone is likely to add a dependency to the go.mod file by hand editing it is when they need to do a "replace" operation to substitute one dependency in place of another (which applies throughout the dependency tree), usually only done when you need to patch a bug in third party code before the upstream is ready to merge that fix.

In my experience, most people either use Dependabot to keep up to date with their dependencies, or they update the dependencies using VS Code to view the go.mod file and click the "buttons"(/links/whatever) that the language server visually adds to the file to let you do the common tasks with a single click. They're both extremely simple to use and help you to update your direct and indirect dependencies.

> But I'm also not assuming they found a perfect solution to package management that all other package management tools failed to find. What's more likely is that they chose a different set of trade-offs, which is what this thread is exploring.

They were extremely late to the party, so they were able to learn from everyone else's mistakes. I really don't think it should be surprising that a latecomer is able to find solutions that other package management tools didn't, because the latecomer has the benefit of hindsight. They went from having some of the worst package management in the industry (effectively none; basically only beating out C and C++... and maybe Python, package management for which has been a nightmare for forever) to having arguably the best. Rust's Cargo comes extremely close, and I've used them both (as well as others) for the past 5+ years in several professional contexts. (Yes, that includes time before Go Modules existed, in the days when there was no real package management built in.) It seems like humans often want to assume that "things probably suck just as much everywhere else, just in different ways, and those people must simply be hiding it", but that's not always the case.

Some "trade-offs":

- For awhile, the `go` command would constantly be modifying your `go.mod` file for you whenever it didn't like what was in there, and that was definitely something I would have chalked up as a "trade-off", but they fixed it. Go will not touch your `go.mod` file unless you explicitly tell it to, which is a huge improvement in the consistency of the user experience.

- Go Modules requires version numbers to start with a "v", which annoys some people, because those people had been using git tags for years to track versions without a "v", so you could argue that's a trade-off too.

- There has been some debate about the way that major versions are implemented, since it requires you to change the name of the package for each major version increment. (By appending “/v2”, “/v3”, etc to the package name. The justification for this was to allow you to import multiple major versions of the same package into your dependency graph — and even the same file — without conflict.)

- The fact that the packages are still named after URLs is a source of consternation for some people, but it's only annoying in practice when you need to move the package from one organization to another or from one domain name to another. It's simply not an issue the rest of the time. Some people are also understandably confused into thinking that third party Go modules can vanish at any time because they're coming from URLs, but there is a transparent, immutable package proxy enabled by default that keeps copies of all[#] versions of all public dependencies that are fetched, so even if the original repo is deleted, the dependency will generally still continue to work indefinitely, and the lock sums will be retained indefinitely to prevent any malicious updates to existing dependency versions, which means that tampering is prevented both locally by your go.sum file and remotely by the proxy as an extra layer of protection for new projects that don't have a go.sum file yet. It is possible to disable this proxy (in whole or in part) or self-host your own if desired, but... I haven't encountered any use case that would dictate either. ([#]: There are a handful of rare exceptions involving packages without proper licenses which will only be cached for short periods of time plus the usual DMCA takedown notices that affect all "immutable" package registries, from what I understand.)

Beyond that... I don't know of any trade-offs. Seriously. I have taken the time to think through this and list what I could come up with above. A "trade-off" implies that some decision they made has known pros and cons. What are the cons? Maybe they could provide some "nicer" commands like `go mod update` to update all your dependencies instead of the somewhat obtuse `go get -u ./...` command? You have complained in several places about how Go combines indirect dependencies into the `go.mod` file, but... how is that not objectively better? Dependencies are dependencies. They all matter, and hiding some of them doesn't actually help anything except maybe aesthetics? I would love to know how making some of them exist exclusively in the lock file helps at all. Before Go Modules, I was always fine with that because I had never experienced anything better, but now I have.

There are plenty of things I would happily criticize about Go these days, but package management isn't one of them... it is simply stellar these days. It definitely did go through some growing pains, as any existing language adopting a new package manager would, and the drama that resulted from such a decision left a bitter taste in the mouth of some people, but I don't believe that bitterness was technical as much as it was a result of poor transparency from the core team with the community.

[0]: https://news.ycombinator.com/item?id=30870862

[1]: https://github.com/caddyserver/caddy/blob/master/go.mod

[munificent]

> You already know[0] the answer to the most of that question, so I don't know why you're asking that part again.

My point is that once you start editing (either by hand or through a tool) the version numbers in the second section, then that's no longer cached state derived entirely from the go.mod files of your dependencies which can be regenerated from scratch. It contains some human-authored decisions and regenerating that section without care will drop information on the floor.

Imagine you:

1. Add a dependency on foo, which depends on bar 1.1.

2. Decide to update the version of bar to 1.2.

3. Commit.

4. Weeks later, remove the dependency on foo and tweak some other dependencies. Tidy your go.mod file.

5. Change your mind and re-add the dependency on foo.

At this point, if you look at the diff of your go.mod file, you see that the indirect dependency on bar has changed from 1.2 to 1.1. Is that because:

A. You made a mistake and accidentally lost a deliberate upgrade to that version and you should revert that line.

B. It's a correct downgrade because your other dependency changes which have a shared dependency on bar no longer need 1.2 and it is correctly giving you the minimum version 1.1.

Maybe the answer here is that even when you ask the tool to remove unneeded transitive dependencies, it won't roll back to an earlier version of bar? So it will keep it at 1.2?

With other package management tools, this is obvious: Any hand-authored intent to pin versions—including for transitive dependencies—lives in one file, and all the state derived from that is in another.

> In my experience, most people either use Dependabot to keep up to date with their dependencies, or they update the dependencies using VS Code to view the go.mod file and click the "buttons"(/links/whatever) that the language server visually adds to the file to let you do the common tasks with a single click. They're both extremely simple to use and help you to update your direct and indirect dependencies.

This sounds like you more or less get the same results as you would in other package managers, but with extra steps.

I don't know. I guess I just don't understand the system well enough.

[skybrian]

As I understand it, that's true in the simple case. If you have `my_app -> foo -> bar` then there's only one path to bar, and you only get a new bar when you upgrade foo.

It's more complicated in general, with diamond dependencies. There needs to be a chain of module updates between you and foo, with the minimum case being a chain of length one where you specify the version of foo directly.

So, people do need to pay attention to security patch announcements. But popular modules, at least, are likely to be get updated relatively quickly, because only one side of a diamond dependency needs to notice and do a release.

[coder543]

> As I understand it, that's true in the simple case. If you have `my_app -> foo -> bar` then there's only one path to bar, and you only get a new bar when you upgrade foo.

This is not correct. You can update bar independent of foo directly from the top-level go.mod file in your project.

[skybrian]

Yes, you can do that by adding a direct dependency on foo. I started by talking about when there isn't a direct dependency on foo.

I explicitly talked about having a direct dependency at the end of the second paragraph.

[coder543]

I'm not talking about direct dependencies. Direct and indirect are all listed in the go.mod file. If they aren't listed there, then they aren't in your final binary. If you delete indirect dependencies from the top-level go.mod, your project will fail to compile.

[geodel]

You keep harping on old buggy version where Go has been very clear that it is operator's explicit responsibility to have correct/updated/fixed versions of dependency running.

It specially does not look good in your case considering you work for Google on a different programing language. If you have a clear point to make then compare it with your approach. Instead of making neutral sounding arguments when they are not.

[adamc]

Don't drag in ad-hominem attacks. If you want to defend Go's approach, explain why having it be the "operator's explicit responsibility" is a good policy, likely to make apps (in general) more secure. The obvious implication of the example given is that, on average, it will be a mess.