Yeah funny, but npm is not the only development community where this regularly happens.
The Onion article this joke refers to [1] is funny because there is a very clear and obvious reason why the U.S. has far more gun deaths per capita. This doesn't apply for npm.
The analogy in npm is dependency proliferation, along with what appears to be weaker community norms around security. To the extent that either or both of these are true, I think the “no way to prevent this” quip is essentially accurate.
Edit: please explain. What other community has this rate of attacks? It's possible they are just detected or publicized less, too. Please help me understand what you're referring to.
some Pipy packages were also attacked recently. And is even more vulnerable due to many projects using requirements.txt which doesn’t lock sub dependencies
Sure, but the Python community isn't the paragon of software risk management excellence you may think it is.
Both Python and Node users (metaphorically) asked for a loaded revolver... They got a metaphorical high yield thermonuclear device with a large blast radius. (And then they skipped the safety tutorial for the B-83 they just bought.)
I initially read this to mean NPM has a pro-concealed-carry policy, which I don't think it does and I don't think is what you meant.
But... Node's culture does not reward "rational" policies with respect to dependency management in the same way that the US does not reward "rational" policies with respect to gun control *. But US gun control policy is a reflection of the "will of the electorate" -- i.e. there are a lot of Americans who want (or need) to own firearms. In the same way, NPM reflects the culture of high-speed, sili-valley web-devs.
I mention both not to criticize, but to comment it's not the tool that's at fault here, but the users who demanded it evolve the way it did. We moved fast. We broke things. And some of the things that remained broken were sociological: It's easy enough to add PGP/GPG signatures on packages, but whom do you trust? What is the meaning of a signature? Does it mean the signer warrants the package/version is free from defects?
NPM is working as designed. Users wanted the software construction equivalent of a loaded revolver. But we got something that was a bit more like a nuclear weapon with a large blast radius. At least the revolver user would more-likely only shoot their own feet six times (or twelve if they reload.)
[*] I'm trying very hard not to start a flame-war about gun control, I only mean to point out dependency management in node can be as contentious in it's domain as gun control policy is in the domain of US politics. Note that I am not making a pro or con argument about gun control, but only pointing out the issue exists. The word "rational" is intentionally chosen to reflect the fact that people's opinions on gun control and package management are often based on personal, often emotional beliefs (which should not be dismissed.)*
> npm is not the only development community where this regularly happens
That is like people defending IIS in the early days by pointing out that Apache occasionally had security problems too. Or, back to the gun control analogy people saying “gun control didn't stop Bondi Beach, did it?” or pointing out [incorrectly]⁴ that everywhere that has gun control has knife crime⁴ instead.
> because there is a very clear and obvious reason why … . This doesn't apply for npm.
I disagree. There are a number of reasons that stack together, the four that spring to the top of my head being:
1. Numbers. There are a lot of potential targets you can exploit if you manage to get something into a ecosystem that large. This “being a big fat target” combined with being easy to exploit makes NPM a very juicy target, and encouraging people to use such a target without trying to implement countermeasures for this sort of attack is IMO reprehensible. Numbers isn't a problem in itself, like in the bad old days when IIS was a mess but Apache got (successfully) attacked far less, but they do exacerbate the security issue by multiplying the attack surface area.
1b. A lot of those using it are relatively untrained or just following recipes so do not know how to protect themselves, and may not even update after an attack like this and remain vulnerable for some further time. While this is not NPM's fault, being due to the popularity/commonality thing, it is something those in control of NPM should care about, if, as I believe is claimed, they care about their users⁰.
2. It is an environment where a ridiculous amount of dependencies, nested impressively deep, is practically encouraged, making audit very difficult even for those who try.
3. A number of good suggestions have been made that would mitigate, or at least vastly reduce, the risks. But action on these has, as far as we know, not happened. Sometimes for good reasons, or at least for reasons¹ rather than “just because”/“cost to implement”/“we don't wanna”, and sometimes, well, not. And no alternatives from within those running NPM are being suggested/worked-on, as far as we know² at least.
4. Those who would make most noise about any change, especially a breaking change that affects them in the smallest way, simply do not care about the risk the situation poses to the wider population.
While the gun control analogy might be a little stretched, I think it is relevant enough particularly because of points 3 & 4.
--------
[0] I refuse to use the word community here. This isn't a cosy little village where everyone knows your name and everyone looks out for everyone else.
[1] Them bringing significant breaking changes, or being too complex to implement piecemeal to give time for those breaks are dealt with or otherwise prepared for, for instance.
[2] If something was being looked into, I'd expect it to be announced³ as that would quieten criticisms like these, at least a little.
[3] Maybe not immediately, but this has been a known problem for so long that we are well past immediately.
[4] When it isn't the case that the US doesn't have knife crime, it just doesn't get reported because the gun issues are worse. Like car travel killing more in total then flights, but you don't hear about every car crash. The UK is often given as an example in these comparisons, but if you look at the stats our knife crime rates are lower than the US's - it isn't that other countries have knife problems instead, the US has worse knife problems as well as the guns problem.
The Node ecosystem happens to be more vulnerable for social and software design reasons, it's true. But people need to be aware that PyPI and Cargo et. al. are not in any fundamental way less vulnerable. This will happen there too.
In fact, attacks like Shai-Hulud explicitly attempt to get into PyPI, and have succeeded to a lesser extent.
But aside from the package-size / -complexity issue pointed out in a sibling comment, PyPI also tries a fair bit to monitor for incoming malware (and there's a "report project as malware" button on each project page).
Also, there are no post-install scripts (of course, the code can detect when it's being run for the first time in the installed environment); and pre-install scripts are only included in sdists[0]. So you can easily[1] configure your installer such that you at least won't get pwned at install time, at the cost that some[2] packages can't be installed that way. And then you can go inspect, run a security scanner over, etc. whatever got installed; wheel installations just copy things to well-defined locations and generate simple wrapper scripts by strict rules.
[0]: I.e., when the project is "being built from source", which generally is only necessary when it includes non-Python code directly and the maintainer hasn't pre-built that code for your system.
[1]: Notwithstanding that, with pip, many actions that you'd expect not to get you pwned totally can. Such as, for example, explicitly telling it to download an sdist and not install it; as I discussed in https://zahlman.github.io/posts/python-packaging-3/ .
[2]: In practice, a pretty small fraction of what typical developers would actually care about, at least outside of specific niches. I'm told there are some niches where it's a big problem, but honestly they're lucky that this kind of build-install orchestration sort-of works at all.
To expand on this, PyPI is slightly less vulnerable because Python users tend to install mega packages (such as numpy or django) and do not frequently interact with their package manager. There is also not a culture of using sub-dependencies.
Cargo is essentially the same as NPM though, it's only "safer" because it's less popular.
Although the situation on NPM is extremely uncomfortable, you're probably less likely to get hit if you take reasonable precautions than on PyPI, simply because NPM is getting scanned more often. Most of these attacks on NPM have been detected and pulled days before my min age kicks in. A sleeper attack on PyPI could be devastating.
> Says Only Development Community Where This Regularly Happens
We've had such issues on other places as well... Shai-Hulud got into Maven [1] and PHP Composer [2], typosquatters got into Maven [3], and it's not new either [4].
No one is safe from skiddies, much less from nation state actors.
> Wake me up when the daily npm security breach headlines are typosquatting stories, not RCE-on-build or RCE-on-upgrade.
RCE-on-build/upgrade can be done in Maven if you manage to compromise one of the major Maven plugins, they run at build time. The thing keeping maven safe for now is that most people pin the plugin and dependency versions, with the obvious side effect that it's truly annoying to get all your dependencies updated.
> The thing keeping maven safe for now is that most people pin [...] versions
Yes, and also the signing of JARs that are uploaded to the repository, and the fact that most release processes are not fully automated, and the batteries-included standard library which reduces the total number of dependencies, and the fact that a run-of-the-mill third-party library can't execute code at build time, and the very small number of people with credentials to publish new versions of major Maven plugins, etc.
NPM by virtue of its popularity and the vagaries of the ecosystem is always going to be a prime target for attackers, and people (maintainers) will always be a weak point.
I agree generally, but I also think it's important to point out that in the NPM ecosystem it is culturally acceptable and even encouraged to install even trivial dependencies to avoid reinventing the wheel. Philosophically I completely agree with this, though practically we see the result and it is not good. The left pad debacle should have been a huge wake up call, but not really much seems to have changed after that.
As long as developers in the ecosystem are cavalier about installing huge chains of dependencies, NPM will be an attractive target for attackers.
Even though we wish it were not so, cultural problems seem to be the hardest technical problems to solve.
Most of these attacks have nothing to do with installing trivial dependencies. It’s usually because the authors npm tokens got hacked; often due to github actions.
The issue is that github actions has too many security gaps that are easy to miss.
No. It is partially due to trivial dependencies. With so many dependencies it is very difficult to evaluate the security posture of all the teams that are inserting themselves into your code.
When I publish commercial software for Unices that use shared object libraries, one of the things we do before publishing is review known vulnerabilities of our 10 dependencies. That is a tractable number. I get a senior engineer to spend time with an intern and step them through the evaluation criteria.
If the team managing a particular library grows lax over time with respect to responding to vulnerabilities, we move away from using that library.
And we can do these things because there are a tractable number of dependencies.
But yes, also GitHub is not pure as the driven slush. I agree with you on that.
Fair point, although when you have dependencies from dozens or hundreds of different publishers, the risk is much higher because it only takes one getting compromised. If instead you only had a handful of core things, there's less surface area
IPCMSes make it somewhat easy to MitM SMS. If your system poops a cookie in the wrong place it doesn't matter if the secret is in someone's head or if it's in a hardware dongle, like you say... the hacker is "in".
My recommendation for bad guys is to not attack the part of the system where it is strong. Just sniff around a bit until you find the weak part and attack that.
Also remember most devs couldn't use a static analysis tool to save their lives (which is why mythos is relevant.) I suspect that a 15 year old copy of Fortify or CoVerity could find bugs mythos missed.
And if that doesn't work, just start scanning github repos for entropy. That's where the credentials that were accidentally published live.
Yeah that’s fair. It’s only as secure as the weakest link.
That was one of the promises of wasm was sandboxing npm packages independently. Not sure what happened with that or not but I’d be curious to know now we’ve had a lot of recent supply-chain publicity.
For example, if every fetched module is sandboxed and even if they got compromised there would be more protection. It would be more “when” not “if” the package is compromised, nip it in the bud.
But then attackers will target the most exposed packages… :)
The "No way to prevent this" analogies seem to me to work better for Memory Safety because, as with Gun Safety, the simple fact is that everybody knows how to solve the problem, but one group insists it's impossible.
There is crowing from the "Actually copy-paste is better" people when this happens, but when it's their turn they just jam their fingers in their ears. The memory safety and gun safety problems are the actual problem. Shai-Hulud would ruin your day if it got into the Odin release you used to build your software, or it was copy-pasted into your "vendor everything" C++ project, the choice not to have automation doesn't mean you fixed the problem.
I'll disagree because the primary issues with gun control in the US are:
[1] Guns are a core part of culture for much of America, very deeply so outside coastal cities. Most of the left wing in the US lives in coastal cities and either grew up there or immigrated very recently and does not leave, so this is an alien concept to them, but even in very blue cities like D.C. you would be shocked how many liberal democrats have armories. It is literally amendment #2!
[2] They are already widely distributed and it would be a logistical impossibility to actually enforce gun control.
This is directly analogous to NPM where:
[1] The package registry working the way it does and people quickly installing packages without thinking much is deeply part of JS culture. It doesn't help that JS caters very heavily to as wide of a market as possible, of which the majority is going to be entry level/junior to associate engineers for whomst script kiddying or letting AI install whatever is essentially a way of life. As evidence, this type of thing is not really a problem with derivatives like Bun, especially in mature organizations where it's easy to enforce a minimum 72 hour wait time between publish and installation of a package.
[2] Packages are already widely distributed and part of dependency stacks (e.g. the infamous leftpad) where it is a logistical impossibility to change how things work.
I also view startups and companies like Vercel as essentially the NRA here, Next.js has taken over huge swathes of the ecosystem and highly encourages dependency-maxxing.
Another direct analogy: proponents of gun control say they are unnecessary for self defense (esp. because law enforcement is good now), too heavy duty to begin with, and fundamentally dangerous.
Similarly I would criticize dependency-maxxing as unnecessary for capability (esp. because AI is good now), too heavy duty to begin with, and fundamentally dangerous.
The whole reason this joke works is because of exactly your belief that somehow you're different and the solution which works for everybody else can't work for you. Charlie is always going to try to kick the ball and Lucy is always going to pull it away and Charlie will never learn from this experience no matter how often it is repeated.
>The "No way to prevent this" analogies seem to me to work better for Memory Safety because, as with Gun Safety, the simple fact is that everybody knows how to solve the problem, but one group insists it's impossible.
I'm not following. Whats the 2nd Amendment equivalent for memory safety? The amount of C/C++ in production or something?
> A well regulated Memory Operation, being necessary to the security of a free Computer, the right of the people to keep and bear Pointer Arithmetic, shall not be infringed.
"A well administered supply chain, being necessary to the freedom of an open internet, the right of the developers to keep and bear hundreds of uninspected transitive dependencies, shall not be infringed."
Language exclusive package managers like this are nightmares for security, but npm simply does things so poorly I feel like they wanted something insecure.
That isn't a language exclusive package manager. I mean things like npm and pip. It isn't necessarily that they're language exclusive, it's that they all tend to have features that aren't good security wise because they aid in developing in that language
The Onion article this joke refers to [1] is funny because there is a very clear and obvious reason why the U.S. has far more gun deaths per capita. This doesn't apply for npm.
[1] https://theonion.com/no-way-to-prevent-this-says-only-nation...