> There is no way for changes in the outside world—such as a new version of a dependency being published—to automatically affect a Go build.
> Unlike most other package managers files, Go modules don’t have a separate list of constraints and a lock file pinning specific versions. The version of every dependency contributing to any Go build is fully determined by the go.mod file of the main module.
I don't know if this was intentional on the author's part, but this reads to me like it's implying that with package managers that do use lockfiles a new version of a dependency can automatically affect a build.
The purpose of a lockfile is to make that false. If you have a valid lockfile, then fetching dependencies is 100% deterministic across machines and the existence of new versions of a package will not affect the build.
It is true that most package managers will automatically update a lockfile if it's incomplete instead of failing with an error. That's a different behavior from Go where it fails if the go.mod is incomplete. I suspect in practice this UX choice doesn't make much of a difference. If you're running CI with an incomplete lockfile, you've already gotten yourself into a weird state. It implies you have committed a dependency change without actually testing it, or that you tested it locally and then went out of your way to discard the lockfile changes.
Either way, I don't see what this has to do with lockfiles as a concept. Unless I'm missing something, go.mod files are lockfiles.
There are some subtleties here, but go.mod files are not lockfiles, including go.mod files don't follow a traditional constraint/lockfile split, which I think is part of the point in that snippet you quoted.
Another way they differ is when installing a top-level tool by default npm does not use the exact version from a library's lockfile to pick the selected version of another library as far I as understand, whereas Go does by default use the exact version required by a library's go.mod file in that scenario.
In other words, go.mod files play a bigger role for libraries than a traditional lockfile does by default for libraries in most other ecosystems.
Here's a good analysis on the contrast between go.mod and the default behavior of more traditional lockfiles (using the npm 'colors' incident as a motivating example):
That link also includes some comments on 'npm ci' and 'shrinkwrap' that I won't repeat here.
All that said, go.mod files do record precise dependency requirements and provide reproducible builds, so it's possible to draw some analogies between go.mod & lockfiles if you want. I just wouldn't say "go.mod files are lockfiles". ;-)
Wouldn't build size increase a lot if transitive dependencies were pinned to direct dependency lockfiles? Like if library A says "use version 1.0.0 of library X" and library B says "use version 1.0.1 of library X", then you'd likely end up bundling duplicate code in your build.
Not saying the tradeoff isn't worth it, but pinning to dependency lockfiles isn't without downsides.
FWIW, that's not what Go does. In your scenario, a Go binary ends up with a single copy of library X -- the 1.0.1 version. That's because library A is stating "I require at least v1.0.0 of X", and library B is stating "I require at least v1.0.1 of X". The minimal version that satisfies both of those requirements is v1.0.1, and that's what ends up in the binary.
That behavior is Go's "Minimal Version Selection" or "MVS". There are many longer descriptions out there, but a concise graphical description I saw recently and like is:
That's the default behavior, but a human can ask for other versions. For example, a consumer of A and B could do 'go get X@latest', or edit their own go.mod file to require X v1.2.3, or do 'go get -u ./...' to update all their direct and indirect dependencies, which would include X in this case, etc.
Continuing that example -- in Go you end up with v1.0.1 of X by default even if v1.0.2 is the latest version of X.
That is a difference with many other package managers that can default to using the latest v1.0.2 of X (even if v1.0.2 was just published) when doing something like installing a command line tool. That default behavior is part of how people installing the 'aws-sdk' tool on a Saturday started immediately experiencing bad behavior due to the deliberate 'colors' npm package sabotage that happened that same Saturday.
In any event, it's certainly reasonable to debate pros and cons of different approaches. I'm mainly trying to clarify the actual behavior & differences.
What if the requirement was pinned specifically to 1.0.0 in order to avoid a bug introduced in 1.0.1. With a package that also requires a minimum 1.0.1, that should be unresolvable set of requirements and your package manager should fail to make a lockfile out of it.
The practice of dependencies’ dependencies being specified using SemVer version constraints to auto-accept minor or patch changes is the difference compared to Go, and why lockfiles will not always save you in the npm ecosystem. That said, approaches like Yarn zero-install can make very explicit the versions installed because they are distributed with the source. Similarly, the default of using npm install is bad because it will update lockfiles, you have to use npm ci or npm install —ci both of which are less well-known.
So it’s not impossible to fix, just a bad choice of defaults for an ecosystem of packages that has security implications about the same as updating your Go (or JS) dependencies automatically and not checking the changes first as part of code review. Blindly following SemVer to update dependencies is bad, from a security perspective, regardless of why or how you’re doing it.
It’s very subtle, but there are some important differences. For example, lockfiles are not recursive in NPM: the NPM package (usually?) does not contain the lockfile and does not adhere to it when installed as a dependency. It will pick the newest version of dependencies that matches the spec in package.json.
Go mod files are used recursively, and rather than try to pick the newest possible version, it will go with the oldest version.
This avoids the node-ipc issue entirely, at least until you update the go.mod.
This really depends on the specific package manager: if you're building an application in Rust, its lockfile will contain the full tree of dependencies, locked to a specific version.
I might be misunderstanding GP, but I think what they're saying is that when package A depends on package B, building package A will use B's lockfile. Assuming that's the case, I think this is generally not how Rust does things, as by default Cargo.lock is explicitly listed in the .gitignore when making a library crate, although there's nothing stopping anyone from just removing that line. I think I remember reading in documentation somewhere that checking in Cargo.lock for libraries is discouraged (hence the policy), but I don't recall exactly where since it's been so long. (That being said, there's a pretty decent chance you were the one who wrote that documentation, so maybe you might remember!)
> I think this is generally not how Rust does things
That is correct.
> as by default Cargo.lock is explicitly listed in the .gitignore when making a library crate
Even if it is included in the contents of the package, Cargo will not use it for the purpose of resolution.
The "don't check it in" thing is related, but not because it will be used if it's included. It's because of the opposite; that way new people who download your package to hack on it will get their own, possibly different Cargo.lock, so you end up testing more versions naturally. Some people dislike this recommendation and include theirs in the package, but that never affects resolution behavior.
The article seemed to go out of its way not to mention any specific package manager or ecosystem. So I think comparing to Rust is completely reasonable.
It didn’t mention one by name, but Rust hasn’t been subject to any widely publicized supply chain attacks. They do, however, mention left-pad by name. I think it can be implied that they really did just mean npm.
I always forget the exact semantics, but the parent's description of them as "recursive" is not the same as Cargo; Cargo determines the full tree and writes out its own lockfile, if dependencies happen to have a Cargo.lock inside the package, it's ignored, not used.
As far as I understand it... the import path that you use to import a package acts as its identity, and only one version of any given package will be installed. The way that it will determine this is by choosing the lowest version specified in any package that depends on a given package. Major versions of packages are required to have different import paths with Go modules, so when depending on two different major versions of the same package, they are treated effectively as their own package.
> by choosing the lowest version specified in any package that depends on a given package
It picks the highest version specified in any of the requirements. (That's the minimal version that simultaneously satisfies each individual requirement, where each individual requirement is saying "I require vX.Y.Z or higher". So if A requires Foo v1.2.3 and B requires Foo v1.2.4, v1.2.4 is selected as the minimal version that satisfies both A and B, and that's true even if v1.2.5 exists).
Okay but if I depend on A and A depends on C and has it pinned at 1.5, but I also depend on B and B has C pinned on 1.8, then I get 1.5? But what happens if C is on 1.8 because it doesn't work with 1.5 because an API it needs doesn't exist in 1.5?
Are we not talking about the transitively pinned dependencies in the "lock" section, or are we talking about logical constraints?
Logical constraints would make more sense, but if the constraints on C across various transitive deps are > 1.5 and > 1.8, and 1.9 or 1.10 exists, I probably want the last version of those.
> Okay but if I depend on A and A depends on C and has it pinned at 1.5, but I also depend on B and B has C pinned on 1.8, then I get 1.5?
No, you end up with the highest explicitly required version. So 1.8 in that scenario, if I followed. (Requiring 1.5 is declaring support for "1.5 or higher". Requiring 1.8 is declaring support for "1.8 or higher". 1.8 satisfies both of those requirements).
> if the constraints on C across various transitive deps are > 1.5 and > 1.8, and 1.9 or 1.10 exists, I probably want the last version of those.
By default, you get 1.8 (for reasons outlined upthread and in the blog post & related links), but you have the option of getting the latest version of C at any time of your choosing (e.g., 'go get C@latest', or 'go get -u ./...' to get latest versions of all dependencies, and so on).
Also, you are using the word "pin". The way it works is that the top-level module in a build has the option to force a particular version of any direct or indirect dependency, but intermediate modules in a build cannot. So as the author of the top-level module, you could force a version of C if you needed to, but for example your dependency B cannot "pin" C in your build.
I'm fairly sure Go Modules does not support what you’re describing. It specifically avoided having a SAT solver (or something similar), unlike most package managers. You specify a minimum version, and that’s it. 1.8 would be selected because it is the highest minimum version out of the options 1.5 and 1.8 that the dependencies require. Unless you edit your go.mod file to require an even higher version, which is an option. Alternatively, you can always replace that transitive dependency with your own fork that fixes the problems, using a “replace” directive in your go.mod file.
If your dependencies are as broken as you’re describing, you’re in for a world of hurt no matter the solution. I also can't remember ever encountering that situation.
If I'm understanding you properly, recursive lockfiles means that if I depend on some chain of dependencies A->B->C->D->E, and E has a security vulnerability that they patch in a new version, I have to wait for A B C D and E to all update their lockfiles before the security vulnerability will be patched on my system?
That's not correct. You can unilaterally decide to update the version of E without waiting for anyone. Alternatively, if only C for example decides to update their required version of E, you would get that version of E if you updated your version of C (directly or indirectly), without needing to directly do anything with E yourself.
But wouldn't that have the same issue then? Developers decide to update their dependencies to patch any security vulnerabilities, and wind up adding installing node-ipc's malicious update
The difference is that it's an explicit choice instead of other package managers who'd happily install latest compromised versions of packages by default.
"A module may have a text file named go.sum in its root directory, alongside its go.mod file. The go.sum file contains cryptographic hashes of the module’s direct and indirect dependencies."
And
"If the go.sum file is not present, or if it doesn’t contain a hash for the downloaded file, the go command may verify the hash using the checksum database, a global source of hashes for publicly available modules."
Should be stressed on. If I committed a dependency version (go.mod) and checksum (go.sum) along with the code, either I get a repeatable build everywhere, or build fails if dependency not found or found to be modified.
I am not sure if all other package managers include checksum with dependency version.
I suspect the primary purpose of the word "may" in that sentence is that you can choose to disable checking the hash against the Certificate Transparency style https://sum.golang.org. In other words, you can opt out. If you do, you fall back to your local go.sum file, which is more-or-less a "TOFU" security model:
https://en.wikipedia.org/wiki/Trust_on_first_use
Discussing "what is a lockfile" is a bit of a headache because different languages have different files which do different things. Generally speaking, there's some file which specifies the dependency versions and some file with cryptographic checksums of the all transitive dependencies.
In Go it's go.mod / go.sum. In NPM, it's package.json / package-lock.json. In Rust it's Cargo.toml / Cargo.lock.
Diving into the exact details of what the author is saying is a bit outside my headspace at the moment. I think the author of the article may not actually understand the scenario where Go's package system differs. (I'm not sure I do, either.)
Suppose you have your project, projectA, and its direct dependency, libB. Then libB has a dependency on libC.
If projectA has a lockfile, you get exactly the same versions of libA and libB. This is true for Go, NPM, and Cargo. However, suppose projectA is a new project. You just created it. In Go, the version of libB that makes it into the lockfile will be the minimum version that libA requires, which means that any new, poisoned version of libB will not transitively affect anything that depends on libA, such as projectA. With NPM, you get the latest version of libB which is compatible with libA--this version may be poisoned.
Conversely, you will get any old security-buggy version of libB instead.
Most package managers when adding a new dependency assume newer versions are "better" than older versions. Go's minimum version system assumes older is better than newer.
I don't think there's any clear argument you can make on first principles for which of those is actually the case. You'd probably have to do an empirical analysis of how often mailicious packages get published versus how often security bug fix versions get published. If the former is more common than the latter, then min version is likely a net positive for security. If the latter is more common than the former, then max version is probably better. You'd probably also have to evaluate the relative harm of malicious versions versus unintended security bugs.
Every change that fixes a security issue implies the existence of a change that introduced the security issue in the first place. Why is bumping a version more likely to remove security issues instead of introduce them?
The reason why older is better than newer has more to do with the fact that the author has actually tested their software with that specific version, and so there's more of a chance that it actually works as they intended.
Security issues aren't introduced intentionally, oftentimes they are found much later on in code that was assumed to be secure. Like the SSL heartbleed vulnerability. Once a vulnerability like that is discovered, you _want_ every developer to update their deps to the most secure version
My statement had nothing to do with intent. Conversely, once a vulnerability is introduced (intentionally or not), you don't want every developer to update their deps to the newly insecure version.
Go just expects you to manually trigger the updates. Thats all. It still is in favor of updating to take security fixes, so i think your argument is wrong.
Let's say my_app uses package foo which uses package bar.
It turns out there is a security bug in bar. The bar maintainers release a patch version that fixes it.
In most package managers, users of my_app can and will get that fix with no work on the part of the author of foo. I'm not very familiar with Go's approach but I thought that unless foo's author puts out a version of foo that bumps its minimum version dependency on bar, my_app won't get the fix. Version solving will continue to select the old buggy version of bar because that's the minimum version the current version of foo permits.
> I thought that unless foo's author puts out a version of foo that bumps its minimum version dependency on bar, my_app won't get the fix. Version solving will continue to select the old buggy version of bar because that's the minimum version the current version of foo permits.
That is incorrect. The application's go.mod defines all dependencies, even indirect ones. Raise the version there, and you raise it for all dependencies. You cannot have one more than one minor version of a dependency in the dependency graph.
As I understand it, that's true in the simple case. If you have `my_app -> foo -> bar` then there's only one path to bar, and you only get a new bar when you upgrade foo.
It's more complicated in general, with diamond dependencies. There needs to be a chain of module updates between you and foo, with the minimum case being a chain of length one where you specify the version of foo directly.
So, people do need to pay attention to security patch announcements. But popular modules, at least, are likely to be get updated relatively quickly, because only one side of a diamond dependency needs to notice and do a release.
You keep harping on old buggy version where Go has been very clear that it is operator's explicit responsibility to have correct/updated/fixed versions of dependency running.
It specially does not look good in your case considering you work for Google on a different programing language. If you have a clear point to make then compare it with your approach. Instead of making neutral sounding arguments when they are not.
The code itself isn’t the only risk factor; it’s weighted by others, like discoverability, which are asymmetric in time. If the white hats fix an issue in version n+1, they’re going to make sure people know. If black hats (or normal devs making a mistake) introduce an issue, no one will tell you about it.
I.e. even if both strategies win just as often, min-version pulls ahead by taking less of a hit from losses.
The author is saying that Go provides the same guarantees with just a package list in the go.mod file that other package managers need both a package list and lock file to solve.
go.sum is essentially a distributed / community maintained transparency log of published versions of packages.
Maybe I'm just not familiar with it enough but I don't see how merging a package manifest and lockfile into a single file is a net win.
This means it's no longer clear which dependencies are immediate and which are transitive. It's not clear which versions are user-authored constraints versus system-authored version selections. For dependencies that are transitive, it's not clear why the dependency is in there and which versions of which other dependencies require it.
Other packages separate these into two files because they are very different sets of information. Maybe Go's minimum version selection makes that not the case, but it still seems user-unfriendly to me to lump immediate and transitive dependencies together.
I think I personally lean towards keeping them in separate files entirely because I like a clearer separation between human-authored content and machine-derived state.
Yes. The go.sum file that sits alongside go.mod keeps track of the hashes so that no modification like that can be made, and dependency fetches actually transparently go through a module proxy/mirror that keeps those same hashes as well, and it will prevent you from getting an altered version of a known module even if you’re starting a new project and don’t have a sum file yet. Versions can’t be republished.
Most package managers have lockfiles. Yes, npm's decision to have both "npm install" and "npm ci", just so you can confuse and mislead developers, is a bit silly.
But Ruby's Bundler, for example, has been refusing to run your code if your lockfile is inconsistent for as long as I remember.
Locking dependencies is, generally, a solved problem across most ecosystems (despite node botching its UX). Go doesn't get to claim that it's superior here.
But of course, supply chain attacks are still possible with lock files. Because somebody is going to update your dependencies at some point (often for security reasons). And at that point you might be pulling in a malicious dependency which you haven't carefully vetted (because nobody has time to vet all their dependencies thoroughly nowadays).
That's still an unsolved problem, as far as I know. I don't think that Go has solved it.
> Locking dependencies is, generally, a solved problem across most ecosystems
As someone who is building a package manager for work, and has looked at pretty much every package manager out there (and their ecosystem adoption), I can only say that those don't reflect the current reality of package management (no matter how much I wish it were true).
Bundler was the first mainstream package manager to adopt a lockfile (AFAIK) a mere 12 years ago. Many many language ecosystems predate that and are still lacking lockfiles (or even widespread adoption of a single compatible package manager).
NPM only got lockfiles 5 years ago (after being pressured by yarn). Gradle got them less than 3.5 years ago, and Maven still doesn't have them (though a niche plugin for it exists). The Python ecosystem is still a hot mess, with ~3 competing solutions (Poetry, Pipenv, Conda), of which Conda just got a 1.0 of their decent conda-lock incubation project a month ago, but due to how setuputils works, the cross-platform package management story is broken almost beyond recovery. In Conan lockfiles are still an experimental feature today.
I could go on and on, but I hope that I could paint a picture that while one could argue that with the advent of lockfiles, locking dependencies has become a solved problem _conceptually_, the current status of implementation across ecosystems is still horrible. I'm also constantly amazed about how little love is put into package managers in most language communities, even though they are so crucial for their respective ecosystems.
As far as I can tell nowadays Go does have one of the better package managers, which given their horrible starting point is quite the feat. As a nice side-effect of experiments in the Go package ecosystem, one of the people working on go dep also created one of the best resources around package managers: https://medium.com/@sdboyer/so-you-want-to-write-a-package-m...
You are probably right that, practically, not every language has "solved" dependency locking and that I was probably unduly extrapolating from my experiences with ecosystems where this has been solved, but there are enough package managers (Bundler, Yarn, Poetry, that I know of, and from what people claim, also Composer and Cargo) that have solved it so that go claiming credit for it seems unwarranted. If anything, this should be credited to Bundler (though it's possible that it wasn't the first package manager to do so).
I can give no credit to npm, it would never have had lockfiles if not for yarn, and even its current attempt seems half-assed.
Python has the problem of too many package managers, some which are bad, unfortunately (the list of open bugs for Pipenv is especially alarming; I once had to rip it out of a project because the dependency resolution failed after half an hour with a stack overflow). That said, poetry solves the problem well and correctly, IMHO.
Over in the Java world, you're right that dependency locking is a bit rarer. But I also think the situation is not nearly as dire there. BOMs, where lists of compatible dependencies are curated, are relatively common, so that alleviates some of the pain. Plus, there seems to be less churn than in some other ecosystems. Still, it would probably be technically better to use dependency locking.
To be fair the title of the article is "How Go Mitigates Supply Chain Attacks" not how it solves.
And I think it does a good job at that.
For example if you had any JavaScript package that depended on node-ipc in your project, a simple npm install after cloning the project would download code that tries to corrupt files in your disk if the malicious code determined that your IP was from Russia. (before the malicious package was taken down/fixed)
With Go you would have to explicitly bump dependency versions. Simply cloning the project and installing dependencies would not have downloaded the malicious version. And bumping would at the very least appear as a diff in a Pull Request.
Yes, node's attempt to include lockfiles is botched, unfortunately. Not only are there UX issues (it's completely unintuitive that you should be using "npm ci", for example), but the lock file can also get corrupted e.g. during a merge conflict and npm performs no sanity checking on it.
I once had a case where a build was suddenly failing. The reason turned out to be that (for whatever reason) a dev had managed to corrupt the lock file, probably during a merge conflict, in such a way that the entry for package A actually contained the URL for package B. It turns out that npm didn't realise that this was inconsistent (with the package.json, and with the npm registry) and downloaded package B but exported it as package A, making the error incredibly hard to pin down.
Doesn't node have lockfiles? Cloning a project and running npm install would install the exact dependencies declared in the lockfile right? To quote the docs[1]:
> The goal of package-lock.json file is to keep track of the exact version of every package that is installed so that a product is 100% reproducible in the same way even if packages are updated by their maintainers.
Maven and gradle don't have lockfiles(by default), and have never really had a serious need for them, because dependency declarations generally don't use ranges.
The central repositories don't allow versions to be replaced, and artifacts are all signed with PGP keys of the developers(although most people don't verify these).
I've never really seen the value in dependency ranges, they make builds more complicated, and bring minimal value.
No it’s the same behaviour for transitive dependencies, if two libraries require different versions of the same transitive dependency, the newer one is chosen.
Deterministic no lock file required.
> if two libraries require different versions of the same transitive dependency, the newer one is chosen.
Unfortunately, this is not how maven works. It picks the version required by the dependency nearest in depth to the project root, breaking ties by first listed in the file. It is deterministic, but it's not what anyone ever wants by default.
(This is called "dependency mediation" if you want to Google it.)
OK, but that has a whole host of other issues such as the dependency that relies on the older version potentially breaking with the newer version. I think BOMs get around this issue, hence why they're so common, but then it just means that you trade one issue for the other and that's why you have different solutions. A Ruby project, for example, doesn't need BOMs.
The elephant in the room here is NPM, and I think the obvious problem there is the culture. I have a tiny app I've been playing with using create-react-app. There are over 800 directories in node_modules. That absolutely dwarfs the number of any other language I've used. Even in a medium sized rails app, you likely have some awareness of what every dependency is. It's just impossible with npm.
One thought I've had to "reboot" the npm culture is to somehow curate packages that are proven to have minimal and safe dependencies, and projects can shift to using those. I imagine there has to be some sort of manual review to make that happen.
In the past I've needed to display a timestamp as something like "n weeks ago" (in a mac app). My first instinct was to write a quick function to do the transform. Then I can tweak it and extend it later to fit my app's needs.
However when I asked the web app team at my company to see their code so I could use the same initial set of intervals, it turns out they use a library to do it. The first instinct of a frontend dev seems to be - even for very tiny, single function solutions - download a library.
To be fair, there are problems that look easy at the first sight, but turn out surprisingly difficult -- and working with time is almost always one of those problems.
npm may be an elephant, but why is it in a room talking about go.mod?
Why not drag APT and the Debian unstable repository in as well? i think install-debian-os also has around 800 dependencies. Debian unstable is vulnerable to supply chain attacks by package maintainers, and Debian stable suffers from lack of volunteers backporting and verifying security patches. Yet, if their numbers increase, the small chance increases a cabal of them attacks the supply chain by pushing and vouching for a fake patch that introduces malicious code ...
The elephant in the room is not some package manager unrelated to the topic. The elephant is trust.
GO argues to mitigate attacks by verifying and locking all dependencies yourself. Which, does not scale.
Debian argues with a proof of work, carefully curating what they publish, and who can publish to fast tracks like the security repo. In theory these people could go insane or be hacked in the same way node module devs go insane or get hacked. But curated publishing mitigates some of the risk.
As far as i know there is only one "third party" GO module repository (actually its generic, supporting both go.mod, npm, and many more) that has a multi-stage system of curation. It promises to integrate lots of tools, global cooperation, configurable policies, an AI and a team of specialists to help with curation. It is also proprietary, stupidly expensive and may not fully deliver on those promises.
The creator of NodeJS talks about how one of the things he regrets is hard-coupling Node to the NPM registry[1]. I imagine this makes it hard to have curated or trusted third-party registries (although note that it is possible to configure private or third-party registries in Node). This is also one of the problems the creator tries to solve in his new runtime, Deno.
I think the NPM organization is completely aware just how dangerous this all is, and is eager to hide it. For example, if you look up an NPM package, it'll list its direct dependencies. But, there's no acknowledgement whatsoever of all the stuff that comes along for the ride.
I'd love to have a well-supported ranking of NPM packages in terms of their dependencies (and their dependencies' dependencies, etc). Knowing the breadth of immediate dependencies, PLUS the depth of the total dependency tree, would give you some inkling of just how much you're taking for granted when you start using a package.
I agree that it would be nice for NPM to show the total footprint of a module, especially if that provides some social incentive to reduce the dependency count.
Something this articles glosses over is that some of these approaches, especially the way 'All builds are “locked”' is achieved with minimum version selection, and “A little copying is better than a little dependency” are tradeoffs against an alternative security model, where transitive dependencies are automatically updated to pick up security fixes.
Part of the churn and noise in the Node.js dependency ecosystem actually stems from security-related issues being noted in a low-level module, and the ripple effects caused by that when a bunch of maintainers have to go around bumping lockfiles and versions.
There is a deeper strategy here with go vs. node; having a standard library maintained by professionals.
I would rather build on a common set of libraries secured by people who are paid full-time to maintain them, and maybe have slightly worse ergonomics, than have a community of libraries that come and go and have inconsistent quality.
This standard library approach yields fewer dependencies, fewer changes over time, and better consistency between projects.
The downside of the standard library approach is that things tend to ossify. While I agree that slower change can be a good thing sometimes, putting things like a HTTP server in the standard library means less experimentation around different ways of doing things, and more difficulty getting performance and other improvements into the hands of language users.
Sure, people can make a third-party module that implements a HTTP server, but the incumbent default that's shipped with the language has an inherent (and often unfair) advantage and a lot of inertia behind it.
I don't really care about the whole "professionals" bit. Sure, I don't want to be relying on something mission-critical to me that's maintained by one person doing it in their spare time. But there is a world of possibilities between that and having a dedicated paid team. Consider, also, that the Go team is only funded so long as Go is important to Google's corporate strategy. Once it isn't, funding will start to dry up, and Go will have to look for a new funding and governance model. That's not necessarily a bad thing, and I'm sure Go would still succeed despite that. But that's kinda my point: this whole "maintained by funded professionals" thing doesn't really matter all that much.
I wish we'd stop trying to make broken languages work. This feels like hill-climbing into the strangest local optimum possible. JS is not the best example of an interpreted language. Wouldn't it be better to put Python in the browser than to put JS on the server? Can't wait for WASM to be a first-rate citizen on the web so we don't have to deal with this anymore.
> Wouldn't it be better to put Python in the browser than to put JS on the server?
I think that's a categorical "no", because Python isn't an objectively better language than JavaScript. I'm saying this as a Python developer since v1.5 (>20 years).
Yes Node.js ships with what is effectively a very thin standard library for some low level things like interacting with the file system, the process model, some security features like TLS.
> tradeoffs against an alternative security model, where transitive dependencies are automatically updated to pick up security fixes.
One thing to keep in mind is that Go doesn't stop you from updating.
For example, its common to do 'go get -u ./...' or 'go get -u=patch ./...' from your project root to update all of your direct and indirect dependencies.
The built-in tooling & language server give you nudges, and if desired it can be automated via things like dependabot or otherwise.
In practice, it means it is often a slightly slower cadence for typical projects in the Go ecosystem compared to say the Node.js ecosystem, but the upgrades still happen. That slightly slower pace I think has worked out so far, and was a conscious choice[1]:
> Many developers recoil at the idea that adding the latest B would not automatically also add the latest C, but if C was just released, there's no guarantee it works in this build. The more conservative position is to avoid using it until the user asks. For comparison, the Go 1.9 go command does not automatically start using Go 1.10 the day Go 1.10 is released. Instead, users are expected to update on their own schedule, so that they can control when they take on the risk of things breaking.
> transitive dependencies are automatically updated to pick up security fixes
Does Node do this? That seems like an awful idea. People should be manually updating dependencies, never automatically. Stuff like dependabot need to die.
I'm annoyed by the false dichotomy that colors most discussions around package management that there are only two solutions to publishing software packages: 1. a carefully curated professionally maintained standard library, 2. the complete wild west where anything goes. It's not really "false" because this is the reality of how package managers are designed today, but it's false in the sense that it doesn't have to be this way.
You can see this tension in virtually every discussion, users resisting using packages that aren't published in the standard library for fear of attacks and poor quality, and maintainers that resist publishing in the standard library for fear of changing requirements and the appearance of better designs. Sure there are admissible entitlement / responsibility arguments against these respective positions, but that's mostly a distraction because both have a valid point.
The problem is that there's no space for intermediate solutions. We need packaging tools to aggregate and publish groups of packages that relate to a particular domain, and organizational tools to ensure quality and continuity of these package groups over time. This mitigates users' fears and reduces their cognitive load by curating the solution space, and it mitigates maintainers fears of ossification and backcompat hell by enabling them to create new package groups.
I'm saying there's an entire dimension of valid tradeoffs in this space, but the current design trend of package managers force us into one extreme or the other.
I'm unclear from reading your comment if you know this or not... but, what the Go team is describing is an intermediate solution. Not the exact one you describe, but it is intermediate. There is no particular requirement to get into the Go package ecosystem, no gatekeepers, it's all namespaced by the URLs you store your source code at, but between what the proxies do and the way the version requirements were specified, you are also not simply naked to every update someone somewhere pushes.
Please note I'm not claiming it's perfect or that you'll like every aspect of it, I'm just saying it is an intermediate solution between the two extremes.
The original solution is the intermediate solution. You can build an entire useful userspace around nothing but libc and an ssl lib (take your pick between OpenSSL or GnuTLS usually). That is effectively what busybox is. Need to do some heavyweight math in Fortran? BLAS and LAPACK probably have everything you need. You can get really far with a C++ application using nothing but Boost.
For whatever reason, newer language ecosystems migrated away from that in the direction of increasingly smaller libraries until npm practically became a parody of it.
There is no reason you can't have lots and lots of useful functionality packed into a few large, well-maintained, well-packaged, well-vetted and trusted libraries, but you need trustworthy organizations willing to that maintaining and vetting. Historically, that seemed to largely be universities and research labs, where the funding and incentives are a lot different from the weekend warriors and solo devs that dominate open source landscapes today. Interestingly, I think library projects that still have large organizations behind them keep with the larger old-school ethos. Look at the world of ML and scientific computing. NumPy and SciPy are still huge libraries. Same with PyTorch and Tensorflow. QuantLib is an interesting example because it actually doesn't have a single large organization behind it. A bunch of Quants just got tired of doing the same things from scratch over and over and decided to aggregate their work for their common good. But it was 22 years ago, so maybe it was still just different back then and the trend toward small libraries hadn't kicked in yet.
I don't think your suggestion works for Web Development.
Imagine an application that talks to a Postgres DB, a Redis cache, several AWS services (e.g. S3, Lex) and also has to be able to parse excel documents. I've worked on applications like that. That's a whole lot of libraries you'll need to include. But I don't think you should include all of them in some "standard library", most people are not gonna need most of these dependencies.
I agree that left-pad is ridiculous (over in the JVM space, there is apache-commons, which also provides e.g. left-pad but of course it's not the only functionality that it provides), but using only 2-3 libraries isn't realistic either.
This is kind of interesting. In the Linux world, you have package maintainers who (among other things) vet and vouch for the quality of the packages they maintain. I think there are similar things in the Docker ecosystem these days (since Docker really did/does seem to be the wild west).
It could be interesting if there was a similar concept for Go (and/or other ecosystems), except that instead of actually packaging the packages into artifacts (especially with the licensing headache that entails), it could be essentially a registry of verified package versions. So the "maintainers" in this sense are just validating the dependencies and maintaining a list of the approved dependencies (including their versions/checksums) and then automated tooling could be used by consumers ("consumer" here may or may not imply payment depending on whether this hypothetical venture is open or closed) to identify unverified dependencies in the consumer's project.
I'm sure someone has thought of this already--link me to relevant projects if you know about any.
> We need packaging tools to aggregate and publish groups of packages that relate to a particular domain, and organizational tools to ensure quality and continuity of these package groups over time
Whats stopping you? golang.org/x is kind of like that. Theres nobody stopping you from aggregating packages under foo.bar domain and build a reputation for high quality.
Good question! The thing that the standard library has that I don't is version aggregation. The problem is not publishing under a single domain, the problem is publishing under a single version. Publishing a bunch of packages that I claim are high quality doesn't help users decide which versions to use, they still have to make this decision on a package-by-package basis. Note that I may be in the middle of a big redesign and some individually-complete packages use the new design but others don't, so you can't just say use the latest of each because they don't all work together yet. In this case I still want to publish each package individually for people who do want fine-grained control, but I wouldn't publish a new version of my "package group" with these versions because they're not integrated yet. My point is that aggregations of verified-interoperable packages is a separate problem domain and existing tools don't suffice to solve it.
You could sort of do that using a Go module that points to all your other modules. Then anyone who depends on that will get the versions you specify (at a minimum).
But a problem is that they would also download all the modules you point at, whether they use them or not.
To fix that, the package system would need a "soft dependency" where, if a module exists, it must be at least the version indicated.
Good idea with "soft dependencies". You'd also need to make the go.mod RequireSpec version argument [1] optional so it can be overridden by the module group.
require example.com/my-module-group 1.2.3
require example.com/some/module // selects minimum version specified by my-module-group
I also noticed Workspaces [2] which I hadn't seen before. They look interesting, but appear to exist for a different purpose. Maybe workspaces with a bunch of replace directives & some cli tooling could emulate a system like what is described here.
The article, and the comments praising this approach, don’t do a great job of explaining how any of this is substantively different from running the likes of yarn install --frozen-lockfile, or cargo build --frozen.
Here’s the thing: You can argue about being secure by default and encouraging better CI practices. I’d fully agree it isn’t great that one has to know a somewhat obscure flag to get a secure CI build in those environments.
But claiming in what I perceive to be in parts a somewhat grandiose tone to have reinvented the wheel, when you’re just describing a standard approach, can make you sound uninformed.
I think at most there's pride in their own solution, which is not something anyone should object to - it's pretty good. It's better than some other systems, but no point in being specific.
Not doing specific comparisons is likely a deliberate strategy, since it means the blog post is less likely to go out of date, and it avoids controversy if they get something wrong.
Comparisons will need to be written by people familiar with both systems, and they're likely to go out of date quickly.
> Unlike most other package managers files, Go modules don’t have a separate list of constraints and a lock file pinning specific versions.
The weird thing about the Go devs is there is always that little bit of elitism under the surface that I detect in their writing (whether it be colors in the playground, the GC, etc). I spent years writing Go and have now moved to Rust. What I find odd is the Rust team has done (IMO) one of the greater achievements in PL history and yet they seem to not have this elitism thing going on (or maybe I just haven't noticed). Go on the other hand, IMO, made some "interesting" language choices (like keeping null) and they seem to want to be celebrated for it and claim their achievements as new and novel.
EDIT: To clarify, I'm talking about the core Go devs - those that work on stdlib and the compiler
I don't see that elitism in this article. Supply chain attacks are a hot topic right now, so it makes sense for them to make a statement about where the language stands with them. They make compelling points, and they're not calling out specific language or package manager as a comparison.
This is a pretty genuinely confounding response, and I mean that with absolutely no offense intended. There is a tremendous amount of fighting between devs who prefer Go and Rust, and a tremendous amount of elitism as well, truly from both perspectives. Rust gained a reputation for elitism long before Go did; “Rust Evangelism Strike Force” was never meant to be pejorative, and “Rewrite it in Rust” was never meant to be a joke, but it became one anyways. It’s not hard to see why; Rust is genuinely novel in a way that few other programming languages are. It feels the most like the “future.”
But I still like Go a lot. I like Go because of how easy and simple it feels. There is definitely elitism over simplicity, but the elitism I’ve seen and even received from Rust and C++ programmers (…despite that I have been coding C++ forever and do have a few Rust projects as well…) has been pretty much the opposite: Go is too stupid and simple; real programmers need absurdly complex metaprogramming to make basic CLI tools or what have you. Now for what it’s worth, that has cooled down in many regards, and also, Rust is amazing and there’s nothing wrong with advanced metaprogramming. (It’s just another set of tradeoffs, after all. Unquestionably has its benefits.)
However, whereas people who have hated on Rust have often come to see it for what it is (an immensely cool, novel programming language,) Go has received the opposite treatment. People soured on it. Now everyone seems sure the GC latency (which of course is just about state of the art) is simply too much for most use cases. It’s seen adoption in all sorts of places and even been competitive with Rust software in performance, but it is commonly discussed as if Go is inherently obsolete because Rust is a better option in every way that matters. Bringing up Go in certain places often subjects you to ridicule, and I’m not joking. The memory ballast is a favorite among detractors to prove that the language is stupid and bad for production environments.
So when people do try to tout the benefits of Go, it’s routinely discredited and downplayed for some reason. It’s a nice language to use with a stellar standard library, nice tooling, and pretty good runtime performance.
This article doesn’t mention Rust (that I noticed) and Go is still being measured up to Rust in the comments. They both trade blows in different categories, but I truly believe that the fact that Go lacks the novelty of Rust with its borrow checker and language design has caused a lot of people to view it very negatively, and I think that is sad. People loved C for a lot of what it didn’t have. Go is a lot different than C, but for me, the sentiment is very much the same.
I think people see what they want to see. I like Go and Rust, but I find myself going back to Go for various reasons and it feels like every year it leads more and more people to ask for justification that they wouldn’t for other languages. It’s a little tiring.
> “Rust Evangelism Strike Force” was never meant to be pejorative, and “Rewrite it in Rust” was never meant to be a joke
Maybe it's just been such a long time, but my recollection was exactly that: both of these terms were invented by outsiders intending to denigrate the Rust community, and became jokes inside the community as a means of recuperating them.
Okay, I will mention one other thing here:
> This article doesn’t mention Rust (that I noticed) and Go is still being measured up to Rust in the comments.
I agree with this in 99.999% of threads, this happens all the time, and probably shouldn't. However, in this thread in particular with the way that the Go package management story developed, including all of the drama there, I don't think it's surprising that Rust/Cargo get mentioned in comparison.
I won’t comment on Rust Evangelism Strike Force too much; it seemed to be unironically used as a term of endearment at some point, but that could’ve been after its use as a pejorative. At this point, I can’t remember, and frankly, the world is probably better off forgetting.
I understand. In the earlier days of Go package management, it was pretty common for folks to compare it to Cargo. In retrospect, this was probably bad, but it did serve to highlight some pretty damning issues with Go at the time. But I feel they addressed the shortcomings significantly with Go modules, and now it has become much more a matter of taste.
I enjoy Go’s idea of trying to make source control the only source of truth, but I don’t think it’s as well-received as the more tried-and-true approach of Cargo and other centralized package repositories. I suppose time will tell.
To be clear, I have no issue with anyone who prefers Go. I was speaking about the core devs in particular. I would agree that the users of the language definitely go back and forth "trading blows".
Interesting and apologies for misunderstanding. I didn’t read the article as being elitist, though I can see how it reads as self-congratulatory to a degree. Maybe the matter-of-fact way that Go’s developers state its advantages comes off poorly compared to, for example, coming from the standpoint of trying to explain how they got to their current design based on the challenges. Personally, I find articles like this easier to read because they tend to be more terse when written this way versus some other approaches that are perhaps more humble.
Lovely comment. I share your sentiment. Go is truly hated on hacker news these days (and, if you're brave enough to venture there, reviled on r/programming).
Go is a distillation of many decades of software engineering experience. The people behind Go (e.g., Russ Cox) have learned from history.
The peanut gallery loves to complain about superficial aspects of Go. Typically these are people with little or no actual experience using the language and tools. They fixate on imagined problems that don't matter in practice.
But anyone who has used Go full-time for a few years is likely to deeply respect and appreciate it.
I've been using Go for a couple of years now and it's funny: I do not love Go, I just work effectively in it. I am not passionate about it, but I recommend it for absolutely all appropriate use-cases.
It doesnt go for intellectual satisfaction, it goes for getting shit done. You have to respect it for being so radically bland.
I haven't used Go full-time, but I have used it on and off for more than a few years. There are certainly things I respect and appreciate about it, but there are also a lot of things that annoy me about it. Some that you might consider "superficial", I consider important. If I'm going to be spending all day in a language, I want the ergonomics of the language itself to work with me, not against me, and Go often does not fit the bill there (for me -- others' opinions are free to differ).
I don't find most of this article to be all that persuasive. Rust, for example, has a separate lock file, which the article derides. That doesn't really make sense, as lock files are also checked into source control, so you get the same benefit that Go touts of go.mod. My threat model doesn't consider having a separate module/package repository to be much of a risk, so I don't care about that point all that much. Admittedly, having source control be the source of truth is just simpler, which is good, but it also means that module publishers can pull versions (or the entire module) for arbitrary, selfish reasons, and then the community is left with a lot of difficulty (there's also a big problem if someone wants to move their code from GitHub to GitLab or something like that). Centralized module repositories can remove this problem if they choose to. The Go Module Mirror appears to be a hack that tacitly admits this problem.
I did find the "a little copying..." bit to be interesting, and I agree with it. With Rust, pulling in a single dependency tends to pull in many tens of transitive dependencies, which I don't like.
"but it also means that module publishers can pull versions (or the entire module) for arbitrary, selfish reasons, and then the community is left with a lot of difficulty"
By default go get will download the source code into the pgk/mod folder. So if a module is pulled by the author, you can just use your copy of the source to fork it.
That's fine if you've already downloaded it, but doesn't help for people trying to pull it for the first time, either as a direct or transitive dependency.
that's what the proxy infrastructure is for: retracted versions won't be selected by default if you're adding it as a new direct dependency, but if you request a version directly (because you need it as an indirect dependency, you have a fresh machine, etc), it will always return the cached code.
For a decade+, people complained that go lacks generics. Would you say those were all people with no experience fixated on an imaginary problem that doesn't matter, or were the complaints valid?
I would say that it is legitimately annoying to have to copy and paste data structures, but mostly doesn't matter, and the intensity of the complaining does not match the intensity of the problem.
People mostly just dont like the expertise of the go devs.
Some of those "superficial aspects" are really annoying.
For example, Go forced version tags to have a `v` prefix in git repos for their dependency system, which broke a whole host of CI tools that expected plain numeric values for release versions. There's a outsized amount of Go-specific special casing for this one seemingly arbitrary decision in multi-language CI systems.
> Go forced version tags to have a `v` prefix in git repos for their dependency system
"Forced" is a bit strong - you can pin any ref, the vX tags just also have some default semver-ish treatment.
> a whole host of CI tools that expected plain numeric values for release versions
Like what? Pure numeric tags are also ambiguous with git refs; this can be worked around by careful arguments, but it means most tooling was already broken when dealing with such things.
If you were one of the people using "release-X.Y.Z" or "rXYZ" I feel for you though.
I've used golang at my employer for several years now, and I came out respecting and appreciating the design decision that have and are going into Java/C#/Kotlin much more given the atrocities I've seen written in golang.
A few years of Golang under my belt and I still hate it. Russ Cox and co. seem incredibly arrogant to me in that they can ignore decades of PL research only to reinvent a bizarre way of achieving what other languages do in a more standard way (package management, error handling), or just adopt that standard super late (generics).
Go has some great qualities and you can make great software with it no doubt. But I find the development of the language frustrating to witness, not inspiring.
C# probably. Though I enjoy Typescript and Python as well.
C# has generics, exceptions, is the birthplace of async/await, has LINQ, an unmatched standard library, a great build system and package management system, is open source, cross platform, fast..
It doesn't compile to a single native binary unlike golang which is a bummer. But 95% of my software ships as a container so this isn't too big of deal for me. The MS-provided base images are really good too.
Agreed. There are some folks very closely associated with other language ecosystems but they spend more time in endlessly critiquing every little thing about Go.
Dependencies being immutable and identified by hash was such an obvious thing 20 years ago. The problem was fitting that into the flow of these crappy UN*X based build systems where to do any mundane task you need to fiddle with files and encodings and semi documented folder heirarchies and use a CLI tool to change other stuff that is too cumbersome to encode directly into files / text. The most obvious concrete instance of this being that it's cumbersome to import a dependency by hash (in Java for example, you really want hashes everywhere instead of reverse TLDs) but obviously better tools (such as a structural code editor) solves this perfectly. It's also annoying every time a company takes a nano step asymptotically towards these proper solutions and spams their crap and everyone buys it, but that's to be expected when your field is broken industrially and academically and even in hobbiest communities.
I feel like much of the current software complexity is mostly caused by the pain that is compiling C dependencies and the outdated Unix build and configuration tools.
Having dependencies not change is literally trivial. Just link to them by hash. Async/await is a pragmatist garbage hack, and does not fit in the archetype you are trying to name.
Off topic but: Since I assumed this was about physical supply chain attacks (where someone nefarious will either intercept your package to install custom firmware etc. or even change the physical device in some way) - does anyone know where I could find a good guide on mitigating such attacks?
Worth noting is that, while go build doesn't run arbitrary code, go generate does:
package evil
import "fmt"
// the echo is intentional, in case someone actually tries this for some reason
//go:generate echo rm -rf /
func PretendGood() {
fmt.Println("I am good")
}
When they say fetching and building code doesn't execute it, that's specific to go get and go build. There's no guarantee that every go subcommand is safe. This is pretty obvious if you know how go generate works and it isn't a flaw of the language, but if I were new to go, this is the kind of article I'd read but still not understand exactly what was safe and what wasn't.
One downside of such pinning of versions is that if one of your transitive dependencies has a security vulnerability affecting your package, your package will remain affected until you update.
As I understand it dependencies-of-dependencies are fetched from the config provided by the library that includes the dependency, in the case of a transitive dependency, everyone between you and the vulnerable package needs to update, and they need to do so in order (i.e. if you depend on A, A depends on B, B depends on C, and C fixes a vulnerability, then first B, then A have to update in order for you to pull in the change).
I feel like everyone is beating around the bush here: NPM is a garbage fire, from the interface to the tooling implementation to the theory to the governance. We can talk at each other back and forth about theoretical benefits, and "friction" vs. "usability" or whatever, but NPM has been and continues to be an unmitigated security disaster. The module proxy could have a package takeover a month for the next three years and still not even come close to the ridiculous shit that has happened on NPM.
One problem: unless you have an endless amount of time, money, patience, and...did I mention patience? that just isn't a viable approach for anything but the smallest of hobby projects
So we just don't build GUI applications unless we understand all of the nuances of layout, text rendering, graphics programming, etc sufficient to implement it ourselves (and without the bugs that even domain experts have introduced but which have been found and fixed over time in libraries)? Or maybe we just say "fuck users who speak languages that aren't easily expressed in ASCII"?
There's a reason libraries exist. It's not like they were the default state of computing and no one has tried to write applications without them. On the contrary, we tried to build applications by writing everything ourselves, but that doesn't survive encounters with the real world. And "well if you can't do it yourself, you don't need it" (which may or may not be your argument, I genuinely can't tell) is just technologically regressive ideology.
Software programmer usually need to solve specific task, not worlds problems. If you go too generalized you will need 10x, 100x or 1000x more amount of time and code.
Look at Big tech, this is what they are doing, they employ thousands workers that write millions of lines of code every day, only to make it work for every case in a world. And still can't compete with specialized solution.
> Software programmer usually need to solve specific task, not worlds problems. If you go too generalized you will need 10x, 100x or 1000x more amount of time and code.
Yes, I accept this is true, but your earlier claim was much more specific: that using dependencies at all makes things worse. I gave you a specific example (GUI libraries) but you completely ignored it. How does your 0-dependencies theory survive an encounter with that basic example?
Except KISS is subjective, as are most things. Please don't take this the wrong way, but anyone who touts any one, true "religion" in sw development I have learned to take with 6 grains of salt. There is no one magic way or solution. Everything is trade offs and I've learned this over the last 30 years through trying every new magic elixir that was going to "save us all".
That said, most sw projects are determined by business need. I can't think of a single one I've ever written that hasn't needed dependencies or would have been viable had I not used some. Going to a theory level, there is no reason to think the stdlib is magically immune to the issues of a very popular dependency either. Shun absolutes and make the correct trade offs based on your business goals.
It may not be quite a URL, but it contains the URL. It definitely more than just "sort-of resembles a URL". See how the module path to URL lookup is done here: https://go.dev/ref/mod#vcs-find
I understand your meaning, but that link supports my claims. The package path helps the Go tool infer the actual URL, but it doesn't contain the URL itself (e.g., the actual URL has a scheme/protocol component and potentially a `go-get=1` query string argument which don't exist as part of the path). This is what I meant when I said it "sort-of resembles a URL", but I understand from this conversation how that wording wasn't clear.
Fair. Strictly speaking it doesn't even "contain" a URL. But I think in the context of this conversation it acts like a URL -- it allows the Go tooling to fetch code from an arbitrary domain and path on the internet.
I guess that's fine so long as you're covered by the standard library and/or are willing to reimplement a lot of stuff yourself, but that's a significant trade-off you're asking.
It sounds defensive but Go stdlib is all you need. I believe I'm qualified to say that as last year I challenged myself to only use stdlib, out of several languages I used over the course of the year on big projects, Go was painless and that was a unique experience. So far this year I haven't seen much need to add libraries to my work because everything is already within grasp.
>> The only commands that will change the go.mod (and therefore the build) are go get and go mod tidy. These commands are not expected to be run automatically or in CI, so changes to dependency trees must be made deliberately and have the opportunity to go through code review.
GO doesn't do jack shit to mitigate supply chain attacks. Version pinning with checksum and that is it. But what could go do? Solve supply chain attacks as a language feature? That doesn't even make sense.
Application developers using Go must prevent supply chain attacks against their applications. So go get some SAST for your pipeline.
Sure there is truth in saying: always verify your dependencies (and their dependencies) yourself with a code review on every update. But you should not do that alone, so let's talk about collaborative vulnerability management. (there is more to sast than vulnerability assessment, but we have to start somewhere)
Let's say repositories that publishe go modules should also publish a curated list of known vulnerabilities (including known supply chain attacks) for the modules they publish. This curation is work: reports must be verified before being included in the list and they must be verified quickly. This work scales with the number of packages published. And worse, modules could be published in more than one repository, module publishing repositories can be different from the modules source code repository, and vulnerability lists can exist independent from these repositories - so reports should be synced between different list providers. Different implementations and lack of common standards make this a hard problem. And implicit trust for bulk imports could open the door for takedown attacks.
There is an argument that vulnerability listing should be split from source and module publishing: each focusing on their core responsibility. For supply chain attacks this split in responsibilities also makes it harder for an attacker to both attack suppliers and suppress reports. But for all other issues it increase distance as reports must travel upstream. And it creates perverse incentives, like trying to keep reports exclusive to paying customers.
To pile on the insanity: reports can be wrong. And there are unfixed CVEs that are many years old (well ok maybe not for go... yet). Downstream there are "mitigated" and "wont-fix" classifications for reports about dependencies and many SAST tooling can't parse that for transitive dependencies.
Really, supply chain attacks are the easy case in vulnerability management, because they are so obviously a "must-fix" when detected. (and to please the never update crowd: for a downstream project "fix" can mean not updating a dependency into an attacked version. you are welcome)
Long story short: go get some SAST in your pipelines to defend against supply chain attacks. Don't pretend pinning the version and half-assing a code review when you update them is actually solving supply chain attacks. Don't tell me everyone who uses go can find a sophisticated data bomb or intentional rce in some transitive dependency of some lib they update to a new feature release. And don't give me some "well if its transitive then the lib dev should have." Should have doesn't solve shit.
Tools like these help you sort out supply chain attacks that other people have already found, before you update into them and push them downstream. Collaboration is useful. Sure you are still left with reading the source changes of all dependency update, because who knows, you may be the first one to spot one, but hey, good for you.
node, python, ruby...etc including go modules can lock their dependencies, Go really wins when it can be wrapped in a reasonably-sized one binary, nothing beats that in deployment, for all the others you have to pull in lots of packages into the target system
I don't like the authoritative tone of the article. Especially
given the fact that author "conveniently forgets" about
go mod edit
and
go work
both of which are deliberately designed counter-mitigations, i.e. they exist
to poke small holes in the pin-everything wall.
I agree with the spirit of the message though, the surface is much smaller
with Go and it shows much planning went into that.
> Unlike most other package managers files, Go modules don’t have a separate list of constraints and a lock file pinning specific versions. The version of every dependency contributing to any Go build is fully determined by the go.mod file of the main module.
I don't know if this was intentional on the author's part, but this reads to me like it's implying that with package managers that do use lockfiles a new version of a dependency can automatically affect a build.
The purpose of a lockfile is to make that false. If you have a valid lockfile, then fetching dependencies is 100% deterministic across machines and the existence of new versions of a package will not affect the build.
It is true that most package managers will automatically update a lockfile if it's incomplete instead of failing with an error. That's a different behavior from Go where it fails if the go.mod is incomplete. I suspect in practice this UX choice doesn't make much of a difference. If you're running CI with an incomplete lockfile, you've already gotten yourself into a weird state. It implies you have committed a dependency change without actually testing it, or that you tested it locally and then went out of your way to discard the lockfile changes.
Either way, I don't see what this has to do with lockfiles as a concept. Unless I'm missing something, go.mod files are lockfiles.