Hacker News new | ask | show | jobs
by zeroCalories 810 days ago
Maintainers can't fully review all code that comes in. They don't have the resources. Even if they could give it a good review, a good programmer could probably still sneak stuff in. That's assuming a maintainer wasn't compromised, like in this case. We need a certain level of trust that the contributors are not malicious.
2 comments
Hackbraten 810 days ago
Definitely this.

I’ve been a package maintainer for a decade. I make it a habit to spot check the source code of every update of every upstream package, hoping that if many others do the same, it might make a difference.

But this backdoor? I wouldn’t have been able to spot it to save my life.

link
imiric 810 days ago
This wasn't caused by not reviewing the code of a dependency. This was a core maintainer of xz, who gradually gained trust and control of the project, and was then able to merge changes with little oversight. The failure was in the maintenance of xz, which would of course be much more difficult to catch in dependent projects. Which is why it's so impressive that it was spotted by an OpenSSH user. Not even OpenSSH maintainers noticed this, which points to a failure in their processes as well, to a lesser degree.

I do agree that it's unreasonable to review the code of the entire dependency tree, but reviewing own code thoroughly and direct dependencies casually should be the bare minimum we should expect maintainers to do.

link
Hackbraten 810 days ago
> Not even OpenSSH maintainers noticed this, which points to a failure in their processes as well, to a lesser degree.

The OpenSSH project has nothing to do with xz. The transitive dependency on liblzma was introduced by a patch written by a third party. [1] You can't hold OpenSSH project members accountable for something like this.

[1]: https://bugs.debian.org/778913

link
imiric 810 days ago
Alright, that's fair. But I mentioned them as an example. Surely liblzma is a dependency in many projects, and _none_ of them noticed anything strange, until an end user did?

This is a tragedy of the commons, and we can't place blame on a single project besides xz itself, yet we can all share part of the blame to collectively do better in the future.

link
imiric 810 days ago
One of the primary responsibilities of a maintainer is to ensure the security of the software. If they can't keep up with the pace of development in order to ensure this for their users, then this should be made clear to the community, and a decision should be made about how to proceed. Open source maintenance is an often stressful and thankless role, but this is part of the problem that allowed this to happen. Sure, a sophisticated attacker would be able to fool the eyes of a single tired maintainer, but the chances of that happening are much smaller if there's a stringent high bar of minimum quality, and at least one maintainer understands the code that is being merged in. Change proposals should never be blindly approved, regardless of who they come from.

At the end of the day we have to be able to answer why this happened, and how we can prevent it from happening again. It's not about pointing fingers, but about improving the process.

BTW, there have been several attempts at introducing backdoors in the Linux kernel. Some manage to go through, and perhaps we don't know about others, but many were thwarted due to the extreme vigilance of maintainers. Thankfully so, as everyone is well aware of how critical the project is. I'm not saying that all projects have the resources and visibility of Linux, but clearly vigilance is a requirement for lowering the chances of this happening.

> That's assuming a maintainer wasn't compromised, like in this case.

What makes you say that? Everything I've read about this (e.g. [1]) suggests that this was done by someone who also made valid contributions and gained gradual control of the project, where they were allowed to bypass any checks, if they existed at all. The misplaced trust in external contributions, and the lack of a proper peer review process are precisely what allowed this to happen.

[1]: https://boehs.org/node/everything-i-know-about-the-xz-backdo...

link
zeroCalories 810 days ago
My understanding is that the attacker was the only maintainer of xz, that was trusted by upstream maintainers. They couldn't realistically check his work. The defence against this can't be "do better, volunteer maintainers". Maybe we could have better automated testing and analysis, but OSS is allergic to those.
link
imiric 810 days ago
Sure, I'm not saying this is the only solution, or that it's foolproof. But this should be a wake up call for everyone in the OSS community to do better.

Projects that end up with a single maintainer should raise some flags, and depending on their importance, help and resources should be made available. We've all seen that xkcd, and found it more amusing than scary.

One idea to raise awareness: a service that scans projects on GitHub and elsewhere, and assigns maintenance scores, depending on various factors. The bus factor should be a primary one. Make a scoreboard, badges, integrate it into package managers and IDEs, etc. GitHub itself would be the ideal company to implement this, if they cared about OSS as much as they claim to do.

link
zeroCalories 810 days ago
Okay, so instead of one random Gmail account taking over a critical project, we need two or three?
link