I understand that this is meant to be an eye-popping press release (and implicitly a product spotlight), but some of these claims make me gag.
It's not an attack "on" PyPI, or even an attack at all: someone is just spamming the index with packages. There's no evidence that these packages are being downloaded by anyone at all, or that the person in question has made any serious effort to conceal their attentions (it's all stuffed in the setup script without any obfuscation, as the post says). The executable in question isn't even served through PyPI (for reasons that are unclear to me): it's downloaded by the dropper script. Ironically, serving the binary directly would probably raise fewer red flags.
Supply chain security is important; we should reserve phrases like "aggressive attack" for things that aren't script kiddie spam.
Thankfully, they're not actually being stolen because all the packages were already taken down; they're available for legitimate use again: https://pypi.org/project/colorslib/
I think it's a serious threat, especially with LLMs now because people can make believable packages at scale. Not everyone vets their packages thoroughly
Speaking of LLMs. Since LLMs like to hallucinate every now and then, an LLM could also hallucinate names of packages that it tells people to install. And those packages could in turn have been squatted by malware authors.
And in this way, malicious packages may be unintentionally downloaded by users even when those malicious packages did not yet exist when the LLM was trained. Just because the hallucinated package name was randomly later taken by someone malicious.
I've seen this effect get amplified also when somebody puts a "bad" answer in a public place like StackOverflow. It is possible to have quite a large blast radius from something like this!
You've always been able to make "believable" packages at scale. PyPI doesn't enforce uniqueness: you can crank out malicious near-duplicates of any package you please.
Stack Overflow and Google search results were already doing that though, at massive scale. I agree it changes things somehow, but people not thinking before acting is not a new problem.
> "While being flooded with spam is never good, it gets immediately noticed and mitigated. It's harder for open source projects to spot and stop rare one-offs"
This is the real problem that NPM and other ecosystems face. A determined attacker that is trying to "poison" a popular Open Source package just has to feign as a maintainer long enough to succeed[0].
Defeating these types of attacks will require rethinking how we think about trust of packages.
Projects like Deno are one approach (fork the ecosystem) while projects like Packj (mentioned elsewhere here), Socket.dev, and LunaTrace[1] are taking the other angle (make it harder to install malware).
It's hard to say which approach is better right away. (Probably a hybrid of both, realistically) It's just non-trivial to fix this in one clean swoop. It's messy.
We’ve built Packj [1] to detect packages with install hooks, embedded binary blobs, and other such malicious/risky packages. It performs static/dynamic/metadata analysis to look for "suspicious” attributes.
A number of academic researchers (including us) have studied malware samples from past open-source supply chain attacks and identified code/metadata attributes that make packages vulnerable to such attacks. Packj scans for several such attributes to identify insecure or "weak links" in your software supply chain (e.g., missing or incorrect GitHub repo, very high version number, use of decode+exec, etc.). Full list here: https://github.com/ossillate-inc/packj/blob/main/packj/audit...
It's relative, but I assume it's flagging for certain class of known malicious patterns. There's nothing stopping you from writing malicious python code, but essentially that script will only run while you expect it to in most cases unless it interacts with the OS in some way.
It doesn't make plain Python code you blindly execute any safer, but at least you've explicitly given those packages your trust. I believe this is more geared toward detecting compromises of those packages you have given that trust.
Packages can do weird things like auto-loading into the interpreter (example: [0]). So in a scenario where a malicious package has ended up on your machine, you're a bit screwed whether it's a .so or a .py. I believe that was the point OP was making -- a pure-Python wheel is not really any safer than a wheel with embedded binaries.
It’s like tor though; everyone that’s malicious doesn’t do this but the ratio is much higher so be much more suspicious. Security isn’t about silver bullets, it’s about a compilation of tricks that make malicious things more obvious.
Wonder if that is related to the malware spamming of NPM that I saw something about last night.
Python used to have a "batteries included" philosophy which tried to put most important stuff into the distro, reducing the number of external dependencies any given app needed. They seem to have abandoned that now, leaving us to fend for ourselves against the malware.
"it's amazing what they were able to do with that old pentium linux server that someone forgot in the server room, so much damage! rust is truly the future"
If a payload is native it's potentially more of a problem than a script. If the payload had been c or c++, I wouldn't have been surprised if they noted that either.
We should only refer to Rust when it's included in positive events? How is it not relevant here? It was used to build executables to inject, likely for malicious purposes. Given its newness and all the other hype around it, I'd say it's very relevant.
Why not also be sure to mention what OS was used for the build, and what linker, and what file format, and what model of computer was used, and what its default ui language was set to? What is so special about programming language used that sets it apart from those other factors that could be mentioned?
Ok, here's a better article from CMU's SEI. See "Binary Analysis Without Source Code".
> In general, the layout used by the Rust compiler depends on other factors in memory, so even having two different structs with the exact same size fields does not guarantee that the two will use the same memory layout in the final executable. This could cause difficulty for automated tools that make assumptions about layout and sizes in memory based on the constraints imposed by C. To work around these differences and allow interoperability with C via a foreign function interface, Rust does allow a compiler macro, #[repr(C)] to be placed before a struct to tell the compiler to use the typical C layout. While this is useful, it means that any given program might mix and match representations for memory layout, causing further analysis difficulty. Rust also supports a few other types of layouts including a packed representation that ignores alignment.
> We can see some effects of the above discussion in simple binary-code analysis tools, including the Ghidra software reverse engineering tool suite... Loading the resulting executable into Ghidra 10.2 results in Ghidra incorrectly identifying it as gcc-produced code (instead of rustc, which is based on LLVM). Running Ghidra’s standard analysis and decompilation routine takes an uncharacteristically long time for such a small program, and reports errors in p-code analysis, indicating some error in representing the program in Ghidra’s intermediate representation. The built-in C decompiler then incorrectly attempts to decompile the p-code to a function with about a dozen local variables and proceeds to execute a wide range of pointer arithmetic and bit-level operations, all for this function which returns a reference to a string. Strings themselves are often easy to locate in a C-compiled program; Ghidra includes a string search feature, and even POSIX utilities, such as strings, can dump a list of strings from executables. However, in this case, both Ghidra and strings dump both of the "Hello, World" strings in this program as one long run-on string that runs into error message text.
The software is rarely at fault if you follow up on the stories. If every time a human driver crashed a car it was the lead story in whatever news you consume, you'd likely never drive.
I mean, not really? There’s a lot of legacy stuff written in other languages, but malware authors have realized that people are less skeptical of rust and are actively taking advantage of that fact.
It's still the same story : PyPI still doesn't have a way to automatically detect interactions with the network and the filesystems for the submitted packages. It's a complex thing to do for sure, but that would be a welcome addition, I guess.
PyPI still doesn't have this because no packaging ecosystem does. It's impossible to do in the general case if your packaging schema allows arbitrary code execution, which Python (and Ruby, and NPM, and Cargo, etc.) allow.
The closest thing is pattern/AST matching on the package's source, but trivial obfuscation defeats that. There's also no requirement that a package on PyPI is even uploaded with source (binary wheel-only packages are perfectly acceptable).
This is a little bit too strong, since packaging doesn't require arbitrary code execution. For example, Go doesn't permit arbitrary code execution during `go get`. Now - there have been bugs which permit code execution (like https://github.com/golang/go/issues/22125) but they are treated as security vulnerabilities and bugs.
What I meant by that is that no packaging ecosystem (to my knowledge) runs arbitrary uploaded code to find network activity. Some may do simpler, static analyses, but outright execution for dynamic analysis purposes isn't something I'm aware of any ecosystem doing.
Python, Ruby, et al. are in an even worse position than that baseline, since they have both arbitrary code in the package itself and arbitrary code in the package's definition. But the problem is a universal one!
This seems eminently solvable though. Why can’t every package submission cause some minimal sandboxed docker image to install the package and call the various functions and methods and log all network and disk activity? If anything looks suspicious it would be denied and the submitter would have to appeal it, explaining why the submission is valid. The same applies for NPM and Cargo. I know there is a researcher out there who has retrieved and installed every single pip package to do an analysis, which is a good start. This seems like the kind of thing that wouldn’t even cost all that much, and big corporate users of python would stand to benefit.
For one, because Docker is not a sandbox, and containers are not a strong security boundary[1]. What you really need here is a strongly isolated VM, at which point you're playing cat-and-mouse games with your target: their new incentive is to detect your (extremely detectable) VM, and your job is to make the VM look as "normal" as possible without actually making it behave normally (because this would mean getting exploited). That kind of work has a long and frustrating tail, and it's not particularly fruitful (relative to the other things packaging ecosystems can do to improve package security).
> I know there is a researcher out there who has retrieved and installed every single pip package to do an analysis, which is a good start.
You're probably talking about Moyix, who did indeed downloaded every package on PyPI[2], and unintentionally executed a bunch of arbitrary code on his local machine in the process.
You make some good points. But it still seems to me that, if you used the best available sandboxed VMs for each platform (Windows Sandbox for Windows; FireJail for Linux; VirtualBox with no folder permissions for OSX-- I don't know if these are the best or even good, those were the ones I found from a bit a searching), that you could install and run these packages in an automated way (especially with some GPT3-type help to figure out how to explore and call the important functions) and look for the telltale signs in the network and file access behavior that they are malicious. Even if we grant that this is a long-tailed "cat and mouse" game, then so what? We won't get 100% security, especially against super sophisticated threat actors, but if you could catch 98% or whatever of the typical clumsy supply chain attacks, or super egregious stuff like that NPM package that deleted your whole disk if you were Russian, that would be an incredibly vast improvement over the current state of affairs. Why isn't that worth doing? Why isn't Google or Microsoft at least trying this?
It isn't worth doing because the equation you've supplied doesn't include the effect of catastrophic failure: dynamic analysis lowers the barrier for exploit to a single hypervisor or VM exploit. Catching 98% of spam packages that affect nobody is worth very little when the 2% you don't catch are the ones that do the real damage.
> Why isn't Google or Microsoft at least trying this?
They are: Google and Microsoft both spend (tens of) millions of dollars on hypervisor and VM isolation research each year. It's a huge field.
> What you really need here is a strongly isolated VM,
Simplify, don't use a VM.
Create an isolated network, hook your sacrificial machine up to it, have it install the package. Remotely kill it (network controlled power switch if needed). The machine's hard drive should be hooked up through a network controlled switch of some type. After the sacrificial machine is powered down, reroute the HD so it is connected to a machine that does forensics.
Now you have a clear "before" and "after" situation setup for analysis.
The sacrificial machine's network activity can be monitored by way of whatever switch/router it uses to connect to the Internet.
Well some calls absolutely should invoke network or disk activity, so you would additionally need to define what constitutes good and bad activity for each. Moreover unless the package is a collection of pure functions it would be easy to hide the malware trigger in state that won't be initialized properly by the automated method calls but would be in the standard usage of the package.
yes, SecurityManager was a poor implementation for many reasons, but it's definitely not "impossible" to sandbox downloaded code from the network while having it interact with other existing code, you can do it with typing alone
I'm not sure it's not do-able, actually. What about having an execution sandbox and a way to check the calls made during the execution of the install script for instance?
I worked a few years back on something like this but it went nowhere, but I still believe it would be doable and useful. The only trace I found back is https://wiki.python.org/moin/Testing%20Infrastructure, which contains almost no info...
Smart attackers are already/will add `sleep(SOME_NUMBER_LONGER_THAN_SCAN_SANDBOX_LIFETIME)` before anything that does FS or network access. Not to say that this wouldn't be a welcome addition, but the scanning needs to be understood in the context of the inherent limitations of large scale runtime behavior detection of packages when you have a fixed amount of hardware and time for running those scans.
Namespacing is not a security boundary: it's a usability feature that helps users visually distinguish between packages that share the same name but different owners. I don't think it would meaningfully affect things like package index spam, which this is.
(This is not a reason not to add namespacing; just an observation that it's mostly irrelevant to contexts like this.)
DNS isn't a particularly secure root of trust; Java is somewhat unique among package ecosystems for picking it as their trust anchor.
It also just kicks the can down the road: Amazon is the the easy case with `com.amazon`, but it isn't clear a priori whether you should trust `net.coolguy.importantpackage` or `net.cooldude.importantpackage`. These kinds of trust relationships require external communication of a kind that package indices are not equipped to supply, and should not attempt to solve haphazardly.
> (I note you're part of pypa!)
I am a member of PyPA, but I don't represent anyone's opinions but my own. It's a very loose collection of projects, and it would be incorrect to read a general opinion from mine.
> Amazon is the the easy case with `com.amazon`, but it isn't clear a priori whether you should trust `net.coolguy.importantpackage` or `net.cooldude.importantpackage`
this is a classic example of not letting perfect be the enemy of good
there is no perfect solution, there never will be
piggybacking off of DNS works extremely well for Java and Go (and the tooling is a pleasure to work with)
meanwhile Python continues to be a complete disaster
I like the way golang handled this. Imports are the URL to the resource. No central distribution mechanism at all. In the past few years they implemented a optional catching layer so you a dependencies going offline doesn't necessarily mean that it unavailable anymore.
Let's encrypt solved this by doing a proof of control over the domain name, and in an automated way.
Pypi could do this. Or, they could require that someone demonstrate proof of ownership for a namespace by signing it with a certificate tied to the domain name (so you couldn't claim the com.bigco namespace without having the certs, which you can't get without owning that domain). There could even be signature requirements/proof for each package and/or version uploaded.
I would need to spend money to purchase a domain and some kind of server before I can publish a python module? That doesn't seem right. And I presume I would need to keep paying for it as long as I want my modules available and verified. Attaching required monetary purchases to an open source ecosystem is not a good idea.
Well, in theory you could have a namespace schema that differentiates between user-submitted and organization-submitted packages such that randomdude's would appear as 'public.randomdude.aws' and organization-owned namespaces verified by a DNS record would appear as 'com.amazon.aws'
You could in principle do proof-of-ownership checks like Google does for things like Webmaster Tools, so you’d need to control a domain to have thr corresponding namespace.
It can be if you implement it to be so. Just let people create an allowlist of approved vendors for their organization or project from those namespaces. This handles not having to approve individual packages from trusted entities like Google, Microsoft, etc. Update the list when new vendors are needed. Reuse elsewhere as necessary.
Maybe the list can be hosted on an internal server for other employees to reuse. Hosting all the packages internally is overkill. Trusting the world by default is overkill.
Now "pip install gooogle/package"
"Hey User, gooogle/package is not from a trusted namespace. Did you mean google/package which is similar and trusted? Or would you like to add gooogle to your local trust file?"
The lack of any kind of curated feeds that only lists verified or popular packges is tragedy. There should be a reasonable way of allowing clients to protect themselves from a typo.
It's like NYC's side walks. Compare pedestrian behavior at say SoHo (daylight) and say LES (nighttime). Amazingly enough, the partying and inebrieted pedestrians at night all file politely in the correct bimodal L|R formation. During the day, it's a rather wild and somewhat uncivilized dynamic slalom formation. My theory: Fangs. The night creatures know someone potentially dangerous maybe in the midst.
It's not an attack "on" PyPI, or even an attack at all: someone is just spamming the index with packages. There's no evidence that these packages are being downloaded by anyone at all, or that the person in question has made any serious effort to conceal their attentions (it's all stuffed in the setup script without any obfuscation, as the post says). The executable in question isn't even served through PyPI (for reasons that are unclear to me): it's downloaded by the dropper script. Ironically, serving the binary directly would probably raise fewer red flags.
Supply chain security is important; we should reserve phrases like "aggressive attack" for things that aren't script kiddie spam.