All you need to know about sudo and frankly most other pieces of the Linux userspace is that it is undertested. The commit that added this flaw to sudo claims to fix a parser bug but includes no tests. There is no reason for the author, the reviewer (if there even was such a person), or anyone else to believe that the bug existed or was fixed by this change. The pull request that supposedly fixes this CVE also includes no tests. There is no reason anyone should believe this fix is effective or complete, or that it does not introduce new defects. This is the result of people who stubbornly refuse to practice even the most basic good engineering practices, like testing and code review, while at the same time using the industry's most dangerous high-level language. As long as this type of thing continues, our tools will remain at a very low level of safety, reliability, and correctness.
"All you need to know about sudo and frankly most other pieces of the Linux userspace is that it is undertested"
Fair enough but what do you recommend?
Me, I try to keep people out of my systems that I don't trust. This particular snag needs local access but I will grant you that my web server or other exposed service might provide a local interface.
Instead of throwing your hands up and screaming "crap" you do your risk assessment and attempt to mitigate as best you can. I read a lot of blogs and have a fair amount of logging and analytics lying around the place (and that's just at home).
Fairly recently I found that my wife's car had loose nuts on the front nearside wheel. That was a change to fix a worn tyre for obvious safety reasons but for whatever reason the fixings were not done up properly. I think they were done up finger tight but a distraction caused the mechanic to forget to use a spanner (wrench) to finish the job to spec. The wheel seemed to work fine but you would get a low rumble sound on corners. It was not a trivial to diagnose fault because you had to notice it before failure - I'm a (non chartered) Civ Eng and IT bod but not a mechanic. There is a minimally screwed on plastic cover that stopped the bolts from flying out - not much.
A car wheel is a thing we can all look at and see that the four bolts are not working properly, once you remove the plastic cover and see them wobble.
Now that is what you can do to protect yourself (risk assess, mitigate etc.) However there should also be something that protects "civilians" and I think that is what is missing. I'm not too sure how we do that.
1. Choose the most boring software possible, trust that the process will work as expected and that you're no worse off than anyone else. Update your software regularly
2. Choose the simplest, most robust software possible (Alpine, OpenBSD, etc). In this case, doas instead of sudo. Pray that works better than everyone else or that you get some benefits through obscurity. Still get surprised every so often. Update regularly
Either way, modern software has gotten complex enough that there's few options for the average person
why do you need to have sudo? I'm perfectly fine without it. sudo is maybe useful in the case where others on the system don't need to know the a password to run as someone else (including root) but need to be able to do that anyway.
sudo seems to have gotten a big installation bases through ubuntu and everybody thinks it's normal now, but really for me it's not.
I want to start and stop a couple of systemd services remotely. Currently I expose the command with sudo and run it under a remotely connected user. What are other options?
You don't actually have to be root at all to manage systemd services. If you give the user `org.freedesktop.systemd1.manage-units` in Polkit then they can just run systemctl as their user and it will work.
If you only want to allow specific units the authorizer is passed the unit under `action.lookup("unit")`.
sudo doesn't have to grant full root access to everyone in the group. It can be set up such that certain users only have the ability to run specified commands as root, which is handy for orgs where you might have a group of tier 1 techs that you want to be able to run certain scripts (written by tier 2 or 3, of course) that require root, but you don't trust the engineers enough to have root access to everything.
Use Actually Portable Executable which enables you to compile textbook C programs as unikernels that boot on bare metal, as well as execute natively on all the existing operating systems too (without needing a runtime or interpreter). We've been working hard to democratize ring0 privileges since spectre has made the performance costs of having an operating system too high: https://github.com/jart/cosmopolitan/issues/20#issuecomment-...
Much of the point of suggesting Mirage was to get away from C - my position is that the correctness costs of C's undefined behaviour (as implemented by real-world C compilers), limited testing support, poor dependency management and so on are too high.
One of my goals with Cosmopolitan is to attract developers who build high-level languages. MirageOS depends on OCaml which depends on C. We need a sturdy foundation at the lowest levels that can enable these visions by helping high-level environments be successful. Cosmopolitan can be that foundation. For example, the codebase has 192 test programs. Much focus has been placed on using the Undefined Behavior Sanitizer and Address Sanitizer to vet everything. My past experience was working on projects like TensorFlow and I started security initiatives like Operation Rosehub which together helped us have the highest performance software infrastructure that's safer too.
By runtime I meant to say external runtimes. Your cosmo binaries are statically don't depend on anything except stable kernel interfaces, when you run on mac/nt/linux/bsd. That means it doesn't need to link any .so files like the glibc runtime. Therefore you won't be impacted by things like linux distro versioning and incompatibilities.
Does it have full support for all the kinds of stacks people are running on Linux today? Is there a robust ecosystem of MirageOS users online able to help troubleshoot issues from the simple to the arcane? Is it supported by VPS providers the world over?
Personally, while I doubt these are the case, having never heard of MirageOS before, I'm willing to be proven wrong. However, if the answer to any of these is "no", I have a hard time seeing how "just abandon Linux wholesale for NewShinyThing" is a viable option for more than a tiny subset of users. (Even if they're all "yes", it's still a wildly unrealistic expectation...)
> Does it have full support for all the kinds of stacks people are running on Linux today?
No, of course not; the only thing that supports everything that Linux supports is Linux. But most use cases don't use every part of Linux.
> Is there a robust ecosystem of MirageOS users online able to help troubleshoot issues from the simple to the arcane?
I doubt it, but how do we get to there from here except by more users starting to use it?
> Is it supported by VPS providers the world over?
Yes, since what you build is just a VM image.
> However, if the answer to any of these is "no", I have a hard time seeing how "just abandon Linux wholesale for NewShinyThing" is a viable option for more than a tiny subset of users. (Even if they're all "yes", it's still a wildly unrealistic expectation...)
I don't disagree as such, but I do think that at this point building anything on these insecure foundations is throwing good money after bad / building castles on sand. Probably 95% of the time you build something that serves your present business purposes, accept a certain amount of insecurity, and get on with your life. But it's worth putting a bit of effort into looking for better ways to do things.
> The commit that added this flaw to sudo claims to fix a parser bug but includes no tests. There is no reason for the author, the reviewer (if there even was such a person), or anyone else to believe that the bug existed or was fixed by this change.
"The PR does not include tests" is not the same as "nobody performed any tests" is not the same as "nobody actually noticed a bug".
And of course, it's perfectly reasonable to form beliefs about code from reading it.
I misread the comment I think, but either way, I am not arguing against automated tests, I was just trying to point out that for some fragments of a code base you should absolutely be able to tell correctness by looking at it.
Even if there were basic unit & regression tests, this bug might not have been caught. This bug should have gone through detailed security review and should probably also undergo fuzzing.
You can bet your bottom banana that the GRU, the NSA, Chinese state security, and the mob have all thoroughly fuzzed sudo and are sitting on the results.
It just seems SO EASY to add a test for this problem, literally the relevant test input is one slash by itself, or any string ending in a slash! So simple! If I sent a change like this at work, no matter how trivial, that said it fixed this bug but I didn't send any tests, the reviewer would reject it out of hand or, probably, just silently ignore my change. But that's the real problem here. This program is a monograph. There are no reviewers and there are, consequently, no standards.
It sounds like you'd find great pleasure in writing harnesses for oss-fuzz. Google has found and reported over 25,000 bugs in 375 projects, but they need people like you to help wire up new projects.
Maybe expecting projects that mostly consist of a single guy working on it in their spare time to be "NSA proof" isn't really realistic?
Folk love to bring up "responsibility" and all of that, but you can't really expect people to bear the responsibility of the world on their shoulders for their spare time projects. It's neither realistic nor fair.
Sudo was (maybe still is?) an OpenBSD project. I don't believe it originated there, but their code forms/formed the basis for sudo pretty much everywhere, much like OpenSSH.
Does the project have a testing framework in place? How is the project built and distributed? Can testing be integrated into its CI/CD pipeline? Is there a CI/CD pipeline? Adding frameworks and infrastructure to a project is one of those things that is a lot more work than it appears to be. They should do it, for sure, but I wouldn’t dismiss it as trivial on a decades-old system that was never built for it.
Single persons often have superhuman standards. It is easy to see that most of the best works in math or art were produced by an individual.
Code review can work, but often it doesn't. There are countless examples of projects with "strict" review requirements that have similar issues (whereas qmail only had one).
Writing tests is the important thing, it keeps you honest.
That’s one maintainer, not even full time according to his résumé. What you just described is multiple specialists and some supporting tools, so another way of looking at this is to ask how much value the IT world has gotten from sudo but not contributed back in support.
When something is this widely used, it’s easy to forget that the answer to who should do more isn’t the one person who actually steps up to keep it alive.
Wow he changed two million lines of the sudo codebase over the project history and made 10,548 commits. That's bonkers. Sudo is clearly doing a lot more under the hood than I thought it did. A simple security critical command shouldn't have that much churn. It should arc towards immutability like TeX, which has had like twelve changes in the last 40 years.
OpenBSD is a fabulous project. I've been working on tool called Cosmopolitan which helps Mac/Linux/Windows/FreeBSD developers write software that's compatible with OpenBSD: https://github.com/jart/cosmopolitan/blob/master/libc/sysv/s... As you can see, I've studied these systems a lot and I've got to say that OpenBSD is the closest to the Bell Labs roots I've seen from community distros. It takes a certain degree of judiciousness to maintain that authenticity and the clairvoyance w.r.t sudo should be all the proof we need that OpenBSD is up to something good.
Sidebar: Wonderful post, but what an awful fake loading bar. Every time I switch from the tab / window to something else and switch back to continue reading I'm interrupted by it for no reason.
This response is the “must be aliens” of security. If you remotely think this is the case, ask whether one of the top intelligence agencies in the world would be more likely to attract attention to a deep cover operation many, many years in the making or would invest in making sure that the bug was well concealed so nobody else would be able to use it on, say, .gov servers. If you remember Dual EC_DRBG I know which one I’d bet on…
Sounds like a very smart person. Much of the value in the test is showing that the old revision fails it. If I were fixing this bug, first thing I would do is write the test and make sure the existing code fails. If it doesn't fail, that means I don't understand the problem yet. Then I fix the problem and commit both the fix and the test. I don't just discard the test, because it is valuable to know that in the future nobody re-introduces the exact same flaw (this regularly happens in open source projects).
I agree. Keeping the test around provides a safety net (a bug repellent of sorts) for the next person the comes along. The test also serves as a documented history of the bugs of the past. The problem is that test does not guarantee the injection of future bugs of a different type. I don't think any amount of testing will. That's not to say we shouldn't write test, we should just be aware of the limits of testing.
> people who stubbornly refuse to practice even the most basic good engineering practices, like testing and code review
That is not fair. This software is maintained by Todd Miller, the well known OpenBSD developer, and has both tests and certainly discusses code proposals. Not everyone has to use github or the language de jour to be useful.
I think the design of the relevant code is worse than the lack of relevant test coverage. The problem solved insecurely by the code instead seems like an obvious target for lexical and syntactic analysis (and this has been so since the sixties, I think).
Yes, please at least use re2c when parsing anything more complicated in C. The result is much more readable, and integration costs are pretty low, and you still keep a lot of flexibility in how you structure your code.
Another question is who wants to maintain four decades old GNU C soup? It was written at a different time, with different best practices.
In some point someone will rewrite all GNU/UNIX user land in modern Rust or similar and save the day. Until this happens these kind of incidents will happen yearly.
You can bet your last dollar that if the "right" RedHat or Debian developer rewrote sudo with feature parity, it would be adopted by all major distros in a couple of months. It's the sort of thing nobody really cares about except OpenBSD (which wrote their own).
The problem is feature parity. Most rewrites cannot guarantee that off-the-bat, so they end up struggling to persuade people to switch - why break stuff that works just fine and lose features, in the name of some engineering purity?
That's part of the dark outcome of so many untested features: it makes it easy to cast FUD upon any potential replacements. As we have no black-box test suite that shows `sudo` implements all of these features, so we also cannot have faith that any replacement would. Ideally it should be possible to run sudo's black box tests against any potential replacement. To start with, we need those tests.
> It was written at a different time, with different best practices.
Not that I have any faith in modern "best practices". I'm imagining `grep` written in modern a modern language like JS and I shutter when I think of the hundreds of micro-dependencies like "left-pad" that would need to be downloaded to make it work.
Maybe it's fair to say that systems programming languages are in a better state in terms of modern best practices than scripting languages.
Try https://github.com/BurntSushi/ripgrep – it’s across the board better, and one of the best personal productivity boosts you can make if you use grep frequently.
> Not that I have any faith in modern "best practices". I'm imagining `grep` written in modern a modern language like JS and I shutter when I think of the hundreds of micro-dependencies like "left-pad" that would need to be downloaded to make it work.
You don't have to imagine: `ripgrep` exists and has all of 10 direct dependencies, 4 of which by the same author because they extracted features / systems from ripgrep (or developed them separately from the start) as they were useful on their own e.g.
* regex, for obvious reasons
* ignore to match and apply ignorefiles (gitignore and friends which ripgrep takes in account by default)
* bstr for conventional utf8 strings rather than guaranteed
* grep intends to be essentially "ripgrep as a library"
Just ten days ago on Hacker News, we had a C programmer claiming that “buffer over-runs are a rare class of bugs, and a class of bugs that are (at least on the heap, and often on the stack) trivial to find and fix” [1].
As a bonus, the person who wrote that turned out to have published C code containing multiple exploitable buffer overflows.
Every now and then we all get a glimpse, for a flash of a moment, that the house of cards has already collapsed. Too invested in our current systems to face this truth, we just update and forget about it until the next time.
Yet another vindication for one of my long-standing practices. I try to avoid installing sudo at all cost on my systems because all it does is increase the attack surface.
Despite this, the wisdom of the crowd is that you should never su to root, for ... reasons? Fat fingering is a thing, but if you can accidentally be in a root terminal without realizing it you have done something horribly wrong.
Heck, from a certain point of view if you have someone in the habit of repeatedly typing sudo over and over again then all sudo has really done is open up every single terminal to be a gateway to the nether realm of super user privs. And in this case, more attack surface.
> Despite this, the wisdom of the crowd is that you should never su to root, for ... reasons?
`su` takes the password of the user you're becoming, while `sudo` takes the password (or not) of the user you already are. So using `su` to become root implies that there's a root password that multiple people (well, assuming there's multiple admins on the box) know.
I firmly agree that sharing passwords is bad, however I can't imagine a threat model where password sharing among multiple parties is more risky than giving multiple people sudo. As zests points out, practically there is no difference because anyone with sudo can change the root password, they are already root, so the auditing use cases is moot if the logging system can't distinguish between two users sudo sued into root at the same time.
Having multiple admins that need to be able to administrate a system might seem like another case where sudo simplifies things, but isn't that what the wheel group is for? Yes there is an M:N issue between administrators and root passwords and we all know that reusing root passwords across boxes is just asking to get pwnd, but if admins can already ssh into a user that has wheel on a system then that implies that there is another existing authentication system that surely could be used to provide the password in a centralized manner. There is complexity in such a service, but that is only required for the M:N case and if it has issues then it can be fixed once. Busted sudo? You have to push emergency updates to (checks notes) ... literally every EC2 image on amazon. If admin:server is 1:N then password manager, copy, paste, and cut sudo out of the loop entirely.
> I can't imagine a threat model where password sharing among multiple parties is more risky than giving multiple people sudo
An individual leaves the company and the security / compliance people say all their access to be revoked, which is kind of a headache with shared logins.
The security / compliance people want audit logs of who does what, which is harder to do with shared logins.
I think both of these things are encouraged (maybe required?) by various certifications that companies doing certain things might need.
sudo can be restricted to specific commands, so you can restrict non-root users to do VERY specific actions (ie, the webdev can reload apache2 but not restart or stop it or take any other action).
This means the webdev has the least amount of access necessary for their work without giving them straight up root or using setuid on a script, which can lead to easy bugs (did you check PATH?)
There are huge differences between running with user as sudo and running as root itself. Much more of an audit trail with sudo. You can trace which account ran which command. Sure you can cover your tracks with sudo but it's far harder than with bare root, even more so with shared accounts.
If a system is configured as you describe, that just means that it then effectively has an additional root/admin account, except under a possibly unpredictable name. So security through obscurity, really (if the privileged user name is secret). That might make sense for some small number of setups, but instead it seems to be enormously popular, and often times it is even used in the security-defeating manner of having the "usual" user be privileged.
This is so silly that it's absolutely ludicrous, but I've never known about or used that before....... I can think of all kinds of other permutations of the command that I've used and know but not this one......
su pulls in random set of currently configured PAM modules, due to requiring authentication.
Anyone checked what buggy horrors await in all those modules? And it's not a static set of old trusty modules either. The fresh new complicated stuff is being added, like systemd-homed, and so on.
pam_systemd and pam_systemd_home have by itself the size of all other 46 PAM modules combined (on my Arch system).
Is there any part of the Linux system that systemd hasn't penetrated yet? It controls boot, processes, authentication, time, dns, proxies, etc. The GNOME Foundation unilaterally required all distros adopt Systemd ten years ago without any community support. Yocto makes it exceedingly hard to create embedded devices that don't have Systemd, so it's probably lurking on all your IoT devices.
I mean, it's literally in the name: Pluggable Authentication Module. It's a very good idea to expose a common interface for user authentication to hide the vagaries of the underlying authentication mechanisms.
It's far more useful to explain why or how PAM is bad, because no one (sane) will agree that the idea of PAM is bad.
Module should be implemented as a separate process running under unprivileged user and communication should be done via pipes. It's UNIX-way. If I understand it correctly, currently module is implemented as a shared library executing under root sharing all the memory with other modules and main program. This exposes way too many opportunities to exploit any vulnerability.
How do you measure the sizes of these modules? I counted lines from source code and systemd pam code was a 1k-2k lines of C total and pam is 30k lines of C.
I mean last time I had sshd hanging and randomly losing access to remote machines due to systemd's newest PAM improvements (homed), when I put the process under gdb, it shown dbus stuff, so it's certainly more than 1-2kloc.
No, complexity strikes again. I haven't used sudo in years, preferring to use doas now. Its essential code is less than 500 lines and it does everything I've ever used sudo for, and that includes much more than `sudo <runupdates>`.
$ man doas | wc -l
58
$ man doas.conf | wc -l
101
$ man sudo | wc -l
741
$ man sudoers | wc -l
3254
And a bonus:
$ man sudoers | grep -C1 despair
The sudoers file grammar will be described below in Extended Backus-Naur
Form (EBNF). Don't despair if you are unfamiliar with EBNF; it is fairly
simple, and the definitions below are annotated.
That only accounts for a small subset of sudo's complexity. It's easily 100x more complex than it needs to be to solve this problem. Now compare the two CVE lists:
My reaction to this vulnerability was mild amusement, then later wondering if I should go discredit the inevitable Rust brigade. We don't have to rewrite everything in Rust to get better security. We just have to use simpler tools.
But if sudo were written in Rust, it could have the same level of complexity and not be vulnerable.
Yes, it would still be vulnerable to logic errors, like the last famous sudo bug where you pass -1 as the UID. But it wouldn't be vulnerable to this. (And this isn't the first memory safety bug to be found in sudo.)
Yes, sudo's complexity is useless for 99.99% of its users. But wouldn't it be nice if the result were merely a gross feeling rather than a security hole?
> But if sudo were written in Rust, it could have the same level of complexity and not be vulnerable
I'm puzzled that we don't have a memory-safe ABI (e.g. amd64-safe) and runtime for C so we could just compile things with
clang -safe sudo.c
to avoid memory errors. I'm fine with sudo (or whatever) taking a 60% performance hit to be more reliable - processors are thousands of times faster now than they were in 1980 when sudo was written. If we had a memory-safe ABI for C/C++ in common use its performance overhead could probably be reduced significantly over time due to implementation improvements, and we might see hardware acceleration for it as well.
There are a number of proof-of-concept memory-safe compilers for C using fat pointers, etc., but memory safety hasn't made it into gcc or clang. 64-bit CPUs can help because you can repurpose address bits. Even Pascal (which is largely isometric to C) supported a degree of memory safety via array bounds checking. I believe Ada compilers also support memory safety. PL/I was actually memory safe and is why Multics never had any buffer overflows. Obviously Rust is memory safe, but for a lot of legacy C code it is impractical to rewrite everything in Rust but eminently practical to recompile it with memory safety turned on.
Encoding array bounds into fat pointers doesn't always work without changing code (e.g. code that uses funky casts, code that makes assumptions about data layout).
Also to ship this in a Linux distro you'd need two builds of many packages. Tons of tools would need to be updated to work with the new ABI. It would be a nightmare.
Furthermore, a new fat-pointer ABI would not address lifetime errors like use-after-free, so what would your plan be there? Boehm GC? More complexity, more overhead, more compatibility issues.
So in practice this is not that appealing, which is why we don't do it.
I think the practical issues you describe like rebuilds of packages and so on are very real if we're talking about general adoption. But if we're talking about recompiling a handful of SUID programs which make up a TCB then I think a proposition like that has a lot of merit and can't be easily dismissed.
Any C code that needs changing to deal with fat-pointers is probably already UB in C (or at best, has some implementation-defined behaviour).
That's because the representation of pointers themselves is undefined (so you can't get a valid result by looking at those). Pointer/integer casts (either direction) are implementation defined. And accesses via pointers to anything beyond their bounds is already UB.
And there's already bodies of work within the Rust (and C/C++) communities around the concepts/technologies that would need to be developed to achieve something like a memory-safe UNIX TCB.
> e.g. code that uses funky casts, code that makes assumptions about data layout
Maybe it shouldn't be doing that? Isn't that the whole point of something like a -safe flag? Increase security as you go along. Yes it is going to take time. The best time to plant a tree is 20 years ago, next best is today.
It would not stop these sort of exploits if I’m not mistaken (ok, it could help since rewriting return addresses is not possible I think), but memory errors that cause logic bugs are still possible.
I thought about that. But when using a higher-level language (that comes with a runtime), you need to give privileges to the whole runtime, which arguably has a much bigger attack surface, unless I am mistaken?
The kernel won't allow you to setuid scripts, there is a reason for this: it's very easy to leave glaring security holes while doing so.
Perl had a fix for this called suidperl, which was a wrapper which enabled Taint mode and other strict checks (https://perldoc.perl.org/perlsec). I don't know of any other language or interpreter that go this far for security by default, so Perl might be the most secure language in this regard. However, suidperl was dropped in 5.12.
My main point was that you could rewrite sudo in all sorts of languages, but saying "just rewrite it in Perl" (assuming it worked) isn't a enough justification to make it happen. Nobody is going to re-create their own project in Perl, Rust, etc just to eliminate buffer overflows. If somebody wants sudo in Rust, they'll have to do it themselves, and it still might never replace the original.
>But if sudo were written in Rust, it could have the same level of complexity and not be vulnerable.
This is not true. Complexity breeds bugs, including security bugs, and memory safety doesn't change that. Your example is a good one - here's another: doas once failed to limit the environment variables which are passed to the child process, which could be used to nefariously influence the program running (e.g. with LD_PRELOAD). How would Rust prevent that oversight? It wouldn't.
A simpler program will generally be more secure than a complicated one, no matter what language either is written in. Furthermore, rewriting an established program from one language to another will always introduce more bugs than it fixes, and more severely the more complex the program is. The single best way to improve security is to reduce the attack surface, and the single best way to do that is to reduce the complexity of your system.
"Yes, it would still be vulnerable to logic errors... But it wouldn't be vulnerable to this. "
I think you'll find in disagreeing with the comment on logic errors you just said the same thing the comment did about logic errors.
Also I think the generalization that rewriting an established bit of code in a new language in a secure language is a bit too general. clearly Firefox not only set out to make Rust for this purpose but it's not had an explosion in vulnerabilities with the modules it has replaced. Quite the opposite actually. Nor has every tool or app rewritten become a security failure compared to the original. I do think it's something that can easily be screwed up though, especially if someone rushes through by focusing on functionality duplication instead of building a more secure version of something.
Regardless, both "using a memory safe language results in a more safe program" and "having a minimum attack sufrace results in a more safe program" can be true. There is no need to make it a choice of A or B.
>I think you'll find in disagreeing with the comment on logic errors you just said the same thing the comment did about logic errors.
I think you'll find that my comment explicitly acknowledges this and expands on it with another example. Are we done telling each other to read the things we're writing?
>Firefox not only set out to make Rust for this purpose but it's not had an explosion in vulnerabilities with the modules it has replaced.
You're setting the bar pretty high with an "explosion" of vulnerabilities here. Rust programs have vulnerabilities, including rewrites. They also have other kinds of bugs, often ones which were not present in the code that they're replacing. You need only browse your nearest convenient RiiR bug tracker to find evidence of this.
Let me restate my thesis in mathematical terms. If we presume that 1 in 100 lines of production code has a bug in it, regardless of language (generous, I know), and that 1 in 10 bugs in C programs are memory corruption related, then saving 10% of those bugs by rewriting it in Rust would take a 10,000 line codebase from 100 bugs to 90 bugs. A 1,000 line codebase, still written in C and without the advantage of memory safety, would have only 10.
In today's example, sudo is a caricature of runaway complexity. Rust is often touted as a panacea, but C has very little to do with why sudo is insecure. Sudo is comically overengineered and that level of overengineering has no place in a security context. This is the larger issue that needs to be addressed, not Rust.
> "Furthermore, rewriting an established program from one language to another will always introduce more bugs than it fixes"
Here[1] is a link to a slideshow of a talk on the F# language, with a case study from EON PowerGen company rewriting the core of an application evaluating revenue due from their balancing services contracts nationwide in the UK. It was originally 350,000 lines of C# developed by 8 people in 5 years and incomplete. It was redeveloped by 3 people (2 had never used F# before) in 30,000 lines, complete in 1 year.
They claim zero bugs in the F# redeveloped system (page 29). This example also gets a mention in a Don Syme (F# language designer) talk in 2018[2] with the PowerGen employee in the audience.
The PDF cites a testimonial from Kaggle saying they're moving more and more of their application into F# which is "shorter, easier to read, easier to refactor, and because of strong typing, contains far fewer bugs".
Incidentally, after inspecting doas for a few minutes, I found two near-vulnerability bugs in it.
The first bug lets any user cause doas to read out of bounds of an array, though not in a way that's exploitable.
Well, it's arguably a bug in libc. If you run doas with a completely empty argv (argc = 0, so not even an executable name; the two systems I tried, Linux and macOS, both let you do this), getopt will exit with optind = 1. Then when doas does;
argv += optind;
argc -= optind;
`argc` will become negative, and `argv` will advance past the null terminator. On most OSes, the `argv` array is immediately followed in memory by `environ`, so argv will now point to the list of environment variables.
doas will then dereference argv, and generally act as if you tried to execute a command consisting of the environment variables. However, the environment variables are not secret, and doas doesn't behave any differently than if you just passed the environment variables as normal command-line arguments, so this is not exploitable.
On an OS where argv is not followed by environ or a similar array of character pointers, doas might crash instead, although since it only reads from those pointers rather than writing to them, this still probably wouldn't be exploitable.
The second bug would compromise memory safety if things were slightly different. The bug is in configuration file parsing. Even if it did compromise memory safety, it would not actually be exploitable, because doas normally only parses the trusted systemwide configuration file. It can be asked to parse a configuration file passed on the command line, but it drops privileges before doing so. This is a good example of layered defense, so kudos to doas for that! Still, I thought the bug was worth mentioning.
The bug is a traditional sort of integer overflow. parse.y grows the array of rules with
maxrules *= 2;
but maxrules is an int, so this will eventually overflow if the configuration file is large enough.
However, because maxrules happens to be signed, before doubling produces a smaller-than-expected positive value, it will first produce a negative value. This will then get sign-extended when converting to size_t (assuming 32-bit int and 64-bit size_t), and reallocarray's overflow check will trigger, causing reallocarray to return NULL. doas interprets that as out-of-memory and handles it cleanly.
(On a system where sizeof(int) == sizeof(size_t), things are a bit different, but it will just run out of memory before maxrules gets that high.)
Moral of the story? Well, as I see it:
Simplicity and layered defense, both featured in doas, are both effective ways to avoid vulnerabilities. But guaranteed memory safety, which would require a different implementation language, is also an effective way to avoid vulnerabilities. You aren't forced to pick and choose. Why not demand all three?
Nice finds. I would agree that that's more arguably a bug in libc than in doas, but also note that the startup code for any language has to consider this case. As far as theoretical operating systems are concerned, this is a consequence of the System-V ABI, so any OS compatible with it would have the same issue.
As for the integer overflow case, it's also highly unlikely to be exploitable, even if it were unsigned - the system would have to, as I'm sure you can infer, have tens of millions of rules before this was an issue. It's quite within the realm of reason, in my opinion, to declare this an acceptable trade-off. The rest of your explanation shows that even if this weren't the case, the bug wouldn't be exploitable.
Anyway, I like your comment, but I'd recommend a different moral to this story: in the space of 47 minutes you were able to conduct a reasonably thorough audit on the doas codebase. Wanna give that a shot for sudo now?
A lot of your statements are pretty strong, and imo totally incorrect.
> Complexity breeds bugs, including security bugs, and memory safety doesn't change that.
Yes, memory safety changes that radically.
> A simpler program will generally be more secure than a complicated one, no matter what language either is written in.
Disagree, but the statement is really weak anyways, especially since 'complexity' is an ill-defined term. More features? Cyclomatic?
> urthermore, rewriting an established program from one language to another will always introduce more bugs than it fixes, and more severely the more complex the program is.
Should be obvious to anyone that this isn't true.
> The single best way to improve security is to reduce the attack surface,
>Disagree, but the statement is really weak anyways, especially since 'complexity' is an ill-defined term. More features? Cyclomatic?
I'm not sure of any definition of complexity you could appeal to which makes my argument weak.
>>rewriting an established program from one language to another will always introduce more bugs than it fixes, and more severely the more complex the program is.
>Should be obvious to anyone that this isn't true.
The opposite is painfully obvious: (1) Writing code causes bugs. (2) Rewriting an established project involves writing more code than leaving it would. (3) Writing all of that new code will introduce new bugs which were not present in the original.
Rust would have prevented the -1 as a UUID too, because you would have used a sum type (Rust enums) instead of a sigil there. Its easier, its idiomatic, its more clear, and the compiler knows how to optimize the overhead away a lot of the time.
Well, in this particular case, the special behavior of -1 is baked into the setresuid system call, while sudo thought it was just an ordinary UID. So if you look at one of the Rust operating system projects designed from scratch from-scratch OS designs, it might not have this kind of pitfall. But if you literally just reimplement sudo for existing OSes in Rust – which I think would be a neat project for someone to take on – you’d be at risk of running into it.
It is a user id, but that bug happened because a -1 was being returned as an error code in one place, and then being accidentally passed in another place. The sum type would be used as the “this possibly errors” return type in the first function, making the bug effectively impossible to happen by accident.
> We don't have to rewrite everything in Rust to get better security. We just have to use simpler tools.
People don't add these features for the fun of it; they're present because they solve a specific use case.
I use doas as well; it's a neat little program that covers many of the common use cases, but it doesn't cover all of them. Usually you should use the simplest tool that solves your problem, but sometimes your problem is complex and thus your tool will be complex.
That is not to say that sudo can't perhaps be simpler; the first version was released in 1985 and there are probably things that can be improved, but sudo really isn't written by idiots who just add features because they have nothing better to do.
Adding features because they solve a specific use-case is grossly irresponsible. Solving a specific use-case is only one of many criteria that needs to be met for a feature to be justified. Others include "is it in scope?", "is it a maintenance burden?", "does it make existing features more unreliable?", "will its bugs affect people who don't need it?", "can it be done in a separate tool?" Anyone can come up with a use-case. It's often the maintainer's job to say "no".
If doas doesn't cover your edge case, then the responsible thing is to make a new tool which covers just that, and not to shove your complexity into a critical security component that the other 99% of the userbase doesn't need. Remember Heartbleed? The entire internet shat its pants because of a vulnerability in a feature that no one uses.
Failing to uphold that principle over and over again leads to broken, unreliable, insecure programs. This is why everything is on fire. Not C.
sudoedit is used by many people, and setting a different shell with -s seems like something that would cover a number of edge cases, yes, but writing a new tool just to add "-s" is obviously silly. Nothing in this particular CVE touches on anything that seems particularly obscure to me.
The last major sudo bug was in the PAM code (which lead to the creation of doas), which is something many people don't need, but also something that many others do need.
And writing separate tools would be the equal (or more!) lines of code and an equal amount of bugs in total (or probably more, since people will be reinventing stuff and there will be fewer reviewers per line of code). This isn't reducing complexity, it's just spreading it out.
For what it's worth, PAM is also not invited to my parties, for all of the same reasons as sudo was shown the door. And what people want PAM for is mostly solved with SSH certificates.
>writing separate tools would be the equal (or more!) lines of code and an equal amount of bugs in total
In total, yes, but crucially, not all on your system at the same time.
I agree about your point on complexity, that was my second thought. Especially for such critical pieces of infrastructure. Thank you for mentioning doa, I will have a look.
But the two are pieces of the same puzzle (defence in depth). Ideally, SUID binaries should be formally specified/verified (ada+spark?), though you could still have bugs in the specification.
And I'd argue that if you need more specific features than just `sudo`'s core functionality, you should probably just make your own setuid binary. That still exposes you to making the same mistakes, so better keep the complexity low, and still rely on a memory-safe language. Using proved, lightweight libraries helps getting an implementation correct.
As much as I like C (which is the language I'm most proficient with), it just gives you too many ways to shoot yourself in the foot, and IMO isn't really the best suited for something where:
- performance doesn't really matter
- memory safety, typechecking are critical.
You could always get away with a transpiler for a DSL, or a compiler that injects more checks, but better suited tools are available anyway.
Hi Drew, I agree with everything you said re complexity and rust. What we really need is a modernized C language, tools that help us catch bugs like this, and a better culture of testing and accountability.
I'm curious whether you run OpenBSD, since you mentioned you use doas. Do you have any thoughts on OpenBSD?
Rust is modernized C. You are looking for something that already exists. If C programmers would be looking for tools to help catch bugs like this and a better culture of testing and accountability they would be using Rust.
Yeah, Rust is about the simplest language that guarantees both memory safety + low-level control. Almost all of its complexity comes from having to satisfy both.
I really don't understand why people donvoted this despite it's obvious. I don't align with you on anything else bc I don't hate C++ and Rust but I have to admit Rust and C++ suffer from the same problems.
But I don’t see a point in using systems languages for the usual UNIX tools, they could be rewritten in a more secure language. In the rare case performance is important, there is FFI, but they are usually IO-bound so there is not much point.
I don't run OpenBSD, for reasons that have little to do with security. In my opinon, OpenBSD is not agressive enough at complexity reduction (doas being an outlier, I think doas is quite a good size).
More CVEs can also sometimes be a function of exposure; Joe Random's program probably has no CVEs but that doesn't mean it's more secure than Jane Popular's tool. In this case, however, both sudo and doas have sufficient exposure to estimate their relative security using CVEs. We can also use the CVEs to characterize the kinds of vulnerabilities each has internally, without comparing them to each other. In general, the CVEs that are discovered for sudo are more severe and damning than those discovered for doas, without respect to their relative occurance.
Sure, that makes sense. Would you also consider the re-occurrence of the same class of CVE? Like the environment variable parsing you mentioned before. If there were another CVE on that for doas would you consider it more damning than the first?
Don’t know why this comment is getting downvoted. The code in question is complex and outdated. Improving the code and testing or switching to simpler tools can lead to a pragmatic solution without the political baggage.
I'm curious, is this one implementation of sudo really used everywhere?
I was under the impression that different Linux userspaces sometimes implement these common commands differently. Like "ls" sometimes actually being aliased to a bash script, or maybe BSD having one implementation and Ubuntu another. Is that not the case? Is "sudo" not maintained by an entity like gnu, bsd, etc?
edit - in other words, I always assumed "sudo" was a highly-dependent system-level tool, not just some useful helper binary that is maintained by one independent person.
Nope... sudo is just this sudo in 99.99% of the cases. There are some alternatives, such as *bsd's doas, and others, but all but doas and su are so non-popular and outdated that I would not recommend using them, as they probably have way more security issues.
doas is just OpenBSD. You can install doas from ports on NetBSD or FreeBSD, just like you can install doas on Linux.
OpenBSD dropped sudo from the base OS several years ago. sudo just became too complex, tailored to the feature creep demanded and required (PAM, ugh) by Linux users.
Briefly going through their website (sudo.ws) I am seriously wondering why anyone would want to put some of those features in a privilege management tool.
Todd Miller is a sharp developer and core OpenBSD contributor. I can only imagine the deluge of requests and pressure he faces to expand sudo. There's no end to the crazy stuff corporations demand, especially when it comes to integration--audit, logging, ldap, etc.
> There's no end to the crazy stuff corporations demand, especially when it comes to integration--audit, logging, ldap, etc.
Why should that be of concern to casual home use? Why do parts of a factory have to trickle down into my home? Wouldn't that be like the need to have a cow to drink milk, or a farm to have something to eat instead of a more apt product to buy for a reasonable price and in good quality?
> Like "ls" sometimes actually being aliased to a bash script, or maybe BSD having one implementation and Ubuntu another
It is true that BSD and linux sometimes have different implementations of posix commands.
The vast majority of linux distros are using the same gnu coreutils though. There are alternate implementations (like busybox, among others), but they're not often used in desktop distros.
I'm curious if you have any example of a linux distro that does treat ls so weirdly; that uses anything other gnu coreutils or busybox for it.
They probably didn't mean replace wholesale, but that `ls` in a shell is a wrapper around the underlying coreutils `ls` with some extra flags by default. Eg:
$ (. /etc/os-release; echo "$NAME:$VERSION_ID")
openSUSE Tumbleweed:20210121
$ command -v ls
alias ls='_ls'
$ grep -A6 -B1 '_ls ()' /etc/profile.d/ls.bash
bash|dash|ash)
_ls ()
{
local IFS=' '
command ls $LS_OPTIONS ${1+"$@"}
}
alias ls=_ls
;;
> introduced in July 2011 (commit 8255ed69), and affects all legacy versions from 1.8.2 to 1.8.31p2 and all stable versions from 1.9.0 to 1.9.5p1, in their default configuration.
It looks like this is pretty far reaching. All of my boxes were vulnerable to this before updating today.
Developers: your moment has come at last to humble your local system administrator for wearing those "I read your emails" t-shirts. This is as day zero as day zero gets. Red Hat and Debian published their security announcements just two hours ago at the exact same moment this was posted on Hacker News. It would have been more responsible to keep something this bad under wraps a bit longer. Because all the people who still use things like cpanel virtual hosting are at risk.
CPanel is a web gui for managing Linux systems. It's mainly used to configure and resell apache virtual hosts. Shell accounts is one of the things it manages. These companies normally have like hundreds of customers per server since they charge ~$1/month for hosting. So anyone who pays one dollar a month extra for shell access can compromise a whole lot of people. I tried tweeting at these virtual hosting providers to bring the vulnerability to their attention, but no one's responded.
> introduced in July 2011 (commit 8255ed69), and affects all legacy versions from 1.8.2 to 1.8.31p2 and all stable versions from 1.9.0 to 1.9.5p1, in their default configuration.
It looks like this is pretty far reaching. All of my boxes were vulnerable to this before updating today.
At this point, can somebody recommend linux security best practices on desktop? I already have two user accounts one for important stuff and other for unimportant stuff. I prefer to use sandboxed apps, but flatpaks don't look well maintained compared to official repositories, and often unofficial. Firejail seems quite controversial due to use of UserNS? What do veterans in security recommend for sandboxing of user apps?
Qualys is great! Love their vulnerability reports.
Just want to echo other praise here for doas. It's fantastic, most likely does everything you need it to do, and is secure. Install it and see for yourself!
I hadn't heard of Qualys until today, and am very unhappy with them. They have thrown us all under the bus by releasing details of this vulnerability before updates are available for major distros. (Still no update for Centos 8 at time of writing this, not sure about any others).
- we can defeat ASLR by partially overwriting the function pointer
getenv_fn (which points to the function sudoers_hook_getenv() in the
shared library sudoers.so); and luckily, the beginning of sudoers.so
contains a call to execve() (or execv()):
People use the tools they have available and understand. Rust, at the time this bug was introduced into sudo, was barely a year old, and hadn't even released 0.1 yet.
Also, the sarcasm in your comment really doesn't help your message.
I have been advocating against C since comp.lang.c days, no need for lessons about how to market something that the audience obviously isn't interested into paying attention.
The first systems programming language that would prevented this kind of exploit was written in 1961, 10 years before C was invented.
Firstly, the constant nagging (of the kind appearing here, and often on other HN threads) against C and family are obnoxious. Any language will have both downsides and upsides and picking this one "flaw" in C and then pretending like it's a reason that C absolutely shouldn't be used is absurd.
Secondly, the Sudo code in question seems to be a result of a poor or nonexistent design process (which also seems consistent with Sudo's horrific bloat of features), so this isn't a good example for preaching against C.
Slightly off-topic, but FTR I view C as somewhat crippled and C++ as an easy to misuse Lovecraftian monstrosity (but much more powerful and useful than C), and I suspect Rust's way of forbidding the user from making memory errors is more trouble than it's worth.
On top of risks common to any language (which type systems are starting to work on), pointer arithmetic adds uniquely catastrophic risks that are no longer justified by big gains at runtime. People keep thinking they can get away with it, but we have decades of proof that they’re wrong. We can afford tools that don’t randomly explode.
First, C has been known to never have been a good option for systems programming, and has UNIX adoption to thank for its existence, instead of having faded into the history of systems programming languages.
Quote from C.A.R Hoare at his Turing Award speech in 1981:
"Many years later we asked our customers whether they wished us to provide an option to switch off these checks in the interests of efficiency on production runs. Unanimously, they urged us not to--they already knew how frequently subscript errors occur on production runs where failure to detect them could be disastrous. I note with fear and horror that even in 1980, language designers and users have not learned this lesson. In any respectable branch of engineering, failure to observe such elementary precautions would have long been against the law."
This is what is missing to force C to finally stop being the JavaScript/PHP of systems programming, liability for exploits with hefty sums.
Second, I keep being told that it is possible to write perfectly safe code in C, it is just a matter of using the tools.
From your assertion, it appears the sudo project has some learning to put into place then, because code reviews weren't enough to prevent this exploit.
C++ tries to gently dissuade the user from making memory errors and still often fails. See: unique_ptr, std::move, ctors, dtors, move mechanics, rule of 0/3/5, etc, It's just more complicated, less elegant, and a pain in the ass.
The -fsanitize=address flag is orthogonal to the runtime library. All -fsanitize=address does is each time the compiler uses memory at address x it generates a few extra instructions which check a bit is set at the shadow address ((x >> 3) + 0x7fff8000). I use a simplified runtime intended to support just that in a few hundred lines of code. Overlaps aren't possible since no shadow address can exist outside the range 0x7fff8000 ... 0x100080000000. Even if you use the stock runtime, I'm not convinced local escalation is possible through having the ability to write foo.PID files. I don't understand how that clobbers files like /etc/shadow.
From 2017, there are at least 5 discovered security issues in sudo[1]. It seems a bit too untested for something running in root and interacting with all users.
Wow, this looks bad. Many VPS and shared hosting providers would be directly shellable with this. Even exploits that got you onto a web server with a limited web shell = full root. Alternatively, sites that already have shells from previous script kiddies can be escalated to root too. Not that I would advocate any of this ^_^ But there are many places where local access is required and you rely on permissions to work properly. A program as important as sudo (or wide spread) is not the kind of place you want to see a vuln this severe
Not the thing saving you, as it is sudo invoked as sudoedit that permits entering the bad codepath... A simple symbolic link and you now have sudoedit.
And... macOS looks vulnerable to me
% cd ~ && ln -s /usr/bin/sudo sudoedit && ./sudoedit -s /
Password:
sudoedit: /: not a regular file
As per the advisory it looks vulnerable (sudoedit: and not usage:)
Not sure if you were being sarcastic, but OpenBSD removed sudo several years ago. The native tool is doas: https://man.openbsd.org/doas sudo just became too complex, what with the feature creep demanded by corporate Linux users.
Not sarcastic. SUID root binaries are a prime target for hackers, and even I've used them to gain local privilege escalation in the past. For example authors of VEGA5000 payment terminal made pppd used for GPRS internet connection have a suid root permissions, and pppd can be used to execute other programs just based on the command line arguments. So that was that.
sudo looks quite complicated, just by looking at what it has to parse and validate (sudoers file). I'd rather not have it on the server, and just use custom purpose made static built suid binaries for necessary minimal purposes of privilege granting.
It's not even installed by default on some popular Linux distros. Of course I realize that it's very ubiquitous, regardless of this. And it's probably fine to use on your workstation. But leaving it unattended on some server feels a bit dreadful.
What variant of `doas` do people run as an alternative? I see Duncaen/OpenDoas, slicer69/doas, multiplexd/doas on GitHub. None seem super widely used as judged by watches/stars/forks.
I would say that the concept and implementation in C is inherently insecure. Switching to something less reviewed because there is a sudo vulnerability is not a guarantee that you are now "safer" especially if those ports are not reviewed.
As far as I can say, never ever use slicer69/doas, I've found 3 critical security vulnerabilities in it, the author does not understand C or how it should work in general.
Here are 3 examples if issues I found and the author used misleading commit titles to hide the issues and made excuses saying a clear buffer overflow very similar to the one found in sudo is just "potential":
a few years ago I found a flaw in sshd. because it was impacting a Linux PAM login/auth module I was writing in C. my module should have worked. but it wasnt. because of sshd. it blew me away, given how important that server is. luckily, others must have complained too, and it ended up being fixed in a newer sshd release. but the fact that it made it into a release in the first place, impacting PAM, was scary
on a not-totally-unrelated note, that was also the last C project I wrote, and since then I fell in love with Go and Rust. for systems code, for me, theres no going back. C is scary given the modern threat ecosystem and whats at stake