[pilif]

Something to keep in mind with regards to qmail is that it's extremely feature-poor and it never got features beyond its initial design goal.

This makes it much easier to keep the bugs out, to the point that making software under such constraints is much more similar to traditional construction projects.

I mean: Nobody ever tells you after you have built a bridge that they are now going to upgrade gravity to gravity 2.0 with 100% more pull. And nobody will ever tell you that your bridge will now get a shopping mall in the middle of it where people can purchase products of their favorite brands.

Software starts to break down when it has to be taken above initial design constraints and when there is not enough time to rewrite subsystems (or all of it) but instead when you have to make the abstractions leaky and compromise.

But back to qmail:

qmail itself is so feature-poor that traditionally, nobody was and is actually running qmail. Instead everybody is running "qmail" which is qmail plus some patches. Sometimes home-grown, sometimes taken from third parties.

But more often than not they are unmaintained and very far removed from the high quality standards of the underlying software.

This is the downside. Yes. You have a bug-free core that totally meets its designers (limited) use-case, but in reality nobody is actually running that.

[geocar]

In contrast, almost nobody runs "exim plus some home-grown patches" or "postfix plus some home-grown patches".

Having the correct architecture, and being able to rewrite the subsystems piecemeal means it is possible for users to experiment with new features organically[1]. That's part of why some qmail installations have features not available in any other mail server even today.

Or to put another way, software purity and homogeny isn't a "good thing", but a trade off: You get to share risk with everyone else who chose like you did, but you're also stuck with the same features and risk everyone else has.

I'd choose "feature-poor download, but highest-feature in production" over "a-few-more-features download and limited-ability-to-upgrade" any day.

[1]: If you're curious, some of the experiments I did are briefly mentioned here: https://news.ycombinator.com/item?id=16166530

[vidarh]

I built a webmail system on top of Qmail back in the day, and I loved it. We rewrote component by component as our needs changed, but the beauty of Qmail was that we could rewrite component by component by sticking to very simple contracts between them and/or start with the Qmail components themselves that for the most part are extremely simple.

Our web frontend for example worked on top of a slightly modified Qmail POP3 server that relied on encoding some message state in the filename, so we need only scan the Maildir without opening the files to be able to e.g. get message read state, and various flags. We also added caching of metadata etc. The changes were tiny and self-contained, and added one extra non-standard command to the pop3 server to retrieve a message list with much more data. The design of Qmail let us start with just changing the pop3 server, and later some tweaks to local delivery, and know we could test those programs in isolation.

The flexibility of Qmail even got us to use it as a messaging middleware later coupled with tinydns - all the routing and retry logic made it very convenient and extremely simple to troubleshoot.

[geocar]

That's great. Do you still have details of it?

I did a webmail "based on qmail" as well. It's still running[1] if you're curious.

[1]: http://demo.internetconnection.net/netmail/

[vidarh]

I have an ancient "backup" sitting somewhere. It was a provider called Nameplanet which provided vanity addresses (lastname.[assorted TLDs] for example), and morphed into Global Name Registry when we launched the .name TLD... The webmail system itself was sold to NetIdentity in 2001 or 2002...

It had a few interesting details - we ran it on ReiserFS for the fast/efficient small file support (at the time it really stood out) which made it great for Maildir's.

We also eventually used a small daemon to poll backends for which server a user belonged to, which had a mechanism to let us mark a user as "busy" so that we could balance accounts between backends by marking it as "busy" on both servers, sync the files over, and then mark it as available again without triggering errors anywhere. qmail on our MX's was modified to look up the right server that way.

The biggest changes were the POP modifications I mentioned. The ones I remember off the top of my head were:

* We modified qmail-local and the pop server to append size changes (from writing a new message or deleting one) to a file used to manage quotas. We appended rather than rewrite because it reduced the need for file locking (we took care to do single writes). We'd lock and coalesce the changes when the file got over a certain size.

* qmail-local was also changed to append the message size, and read-status to the filename. That let us avoid stat() calls for the files for the filesize, and opening and reading the files for unread counts etc. It was one of the first optimizations we did.

* Then we added a cache file that contained subject, sender, size, attachment status etc., for the web frontend, which would be dynamically re-generated automatically as needed.

* We made "+[something]" sort directly into folder "something".

* When we sold it I was most of the way through adding Sieve support to our qmail-local replacement.

These changes were quite small, and each successive changes lowered IO load dramatically (we handled about 2 million accounts before it was sold). Today, we could probably handle the IO load and storage we had with a single NVMe card...

The web frontend would try to use our extended POP3 command, and then fall back to scan the messages (and store a cache locally on the frontend) if it wasn't available, so we could use it as a POP3 client for other backends too. (The frontend is a story in itself - C++ CGI statically linked to shorted load time (it made a big difference at the time) and with "delete" only for really large allocations, to avoid wasting time on deallocation since we knew each process would at most live for a few seconds.

[geocar]

Neat!

It's a shame that more people don't develop software with this kind of... change-for-purpose.

[cesarb]

And you need these patches to fix what is IMO qmail's worst problem: backscatter. As far as I remember, when receiving an email with a forged return address to a non-existent mailbox, qmail first accepts the email, then sends a bounce message to the forged return address. Other MTAs (and patched qmail) reject the email directly in the SMTP session, preventing this issue.

I personally consider this backscatter issue a design bug in qmail.

[NoGravitas]

I worked at a web hosting company 1999-2005 that used qmail, and while there were many things wrong with qmail, due to it not being designed for the realities of email circa 2001, backscatter was by far the worst. We were processing significantly more backscatter than valid email, and to the best of my knowledge, the patches to address it didn't yet exist.

We certainly should have switched mail servers, but qmail was deeply ingrained in our home-grown hosting automation system, and it would have been a big deal to change.

[stevekemp]

That was definitely true, and it was the reason that I personally stopped using qmail in a previous (very long time ago now) job.

There were patches to fix the problem, along with offering useful features, but for whatever reason we went with exim (exim 3.x from Debian).

[aplorbust]

"qmail itself is so feature-poor that traditionally, nobody was and is actually running qmail.

Instead everybody is running "qmail" which is qmail plus some patches."

I ran and continue to run qmail without any patches.

The above quoted statements thus cannot be true.

But maybe "nobody" and "everybody" are figures of speech?

[NoGravitas]

So...are you running an open SMTP relay, or are you refusing to relay mail for your own users? Because I'm pretty sure that with an unpatched qmail, you've got to be doing one or the other.

Also, how are you dealing with backscatter?

[aplorbust]

Not using qmail the way you are (incorrectly) assuming.

I am an end user not an email provider.

For example I use qmail to provide "inter-device email" on a local network of devices all belonging to the same user, and not connected to the internet. Not that I love email but these devices are sometimes "locked down" by default and email is one of the few ways to move files between devices without using the internet.

Another example is using qmail on a tap-based layer 2 overlay (not OpenVPN) to provide encrypted "peer-to-peer email". Each peer is running qmail-smtpd bound to a tap device. This was an experiment to prove encrypted email is easy.

qmail running under curvecpserver is another experiment.

[technion]

qmail won't even compile on modern glibc without the errno patches - you must have at least some patching done.

[aplorbust]

Not using glibc.1

Not using Linux.

Advice for all commenters who make presumptions about others computer use: Please kindly check your assumptions.

1 Is this an issue for musl and the various other alternatives to glibc? I have no idea but seems like only referring to glibc 2.3.x and up is a bit myopic. Its possible some users might not be using that library. I am one such user.

[peterwwillis]

I wrote a qmail masquerading plugin when I was 17. Would it have been nice as a feature? Sure. Did my plugin suck? Definitely. But it worked, and the core software stayed secure, while I watched others patch sendmail every year.

[pilif]

> But it worked, and the core software stayed secure

are you sure? How can you be sure that your custom patches didn't affect the security of the core product? qmail wasn't designed to be extensible. It had no plugin interface.

Of course it's possible that you didn't make a mistake back then.

Just as it's possible that I didn't make a mistake when I was 18 and wrote a patch to Cyrus imapd to allow authenticating against an SQL database.

But TBH, when I look back at the code I wrote back then, at least in my case, I'm quite sure I f'ed up in various ways.

Thankfully, I never shared these patches with other people.

[peterwwillis]

Oh I'm sure it was bug ridden. But even if my feature introduced a security hole, you would have to find and exploit it, and it would then have to find a way to attack the rest of the app (which qmail makes difficult).

It's kind of like using OBSD as your app platform. You can definitely make it insecure! But it's more secure by default than others, perhaps because of a lack of features, as well as very good security design.

[irundebian]

Are you sure that you understood djb's statement about the principle of least privilege? It's not about attacking the rest of the app, but about violating the user's security requirements.

[peterwwillis]

I don't understand what this has to do with my comment.

[irundebian]

> it would then have to find a way to attack the rest of the app (which qmail makes difficult).

It's not necessary to attack the rest of the app as soon as user's security requirements are violated. So if an attacker had been able to have an impact on confidentiality, integrity or availability because of your masquerading patch, user's requirements would have been broken. For an impact on availability controlling control flow isn't necessary, you just need to crash components.

[Florin_Andrei]

> nobody was and is actually running qmail

I had Qmail running circa 1998 and I'm pretty sure it was vanilla, no patches. But after all this time I could be wrong.

Anyway, shortly thereafter I discovered Postfix and that was the end of my relationship with Qmail.

[NoGravitas]

In 1998, it was probably reasonable to run qmail without patches. By 2000 or 2001, it definitely wasn't.

[adrianN]

It happens that bridges get more lanes, or more cars.

[pilif]

In most cases that won't require a fundamental change in the architecture though. And when it does, then the bridge is rebuilt in accordance with the new requirements.

[Fnoord]

Ridiculous example: Qmail refused CRLF e-mails because the headers couldn't be read. It only accepted LF e-mails. In practice, this meant an e-mail client like Outlook did not work. It was solved by a very small fix with a few lines of C, but the patch I found did not apply cleanly either because it was for an older version or because of other patches, so I had to port it. This makes something like autoconf where you gotta specify all kind of options to ./configure a breeze.

[geocar]

You have that backwards. Qmail rejected emails with bare LF because it was in violation of the email spec.

https://cr.yp.to/docs/smtplf.html

[pilif]

It's still a real issue though: As an ISP you can't run a mail server that doesn't accept mails from the one client that's most used among your customers.

Yes. Outlook isn't conforming to the spec, but it's also being actively used and even if you could put pressure on Microsoft to fix it (good luck doing that back in the early 00s), you can't possibly force all your customers to update.

Now you have three options:

1. you switch MTA to one that can deal with the non-conforming clients.

2. you add a proxy server that interfaces between the non-conforming clients and your MTA

3. you patch your MTA

Unfortunately, because of qmail's good reputation ("hey! we're running qmail that never had security issues!") and because of the lack of abilities of your run of the mill ISP to write a scaling SMTP proxy, what people have traditionally have done is option 3.

What they forget about option 3 is that the one big advantage of that solution ("hey! we're running qmail") isn't valid any more because you're not exactly running qmail any more. You're running qmail plus some additional patches that actually touch the public interface of your MTA and are thus exposed to the network. To unauthenticated users.

So IMHO, they should have gone with option 1) or, nowadays where it's easier to write a well-scaling SMTP proxy thanks to the raise of asynchronous event based communication, option 2, but you'd better be sure you're not introducing security flaws in your reverse proxy.

[geocar]

It's not a real issue because those clients don't exist anymore.

Nearly twenty years ago when they did, (4) I could use fixcrio (which is hardly a proxy server): I simply ran it one of the ports that accepted mail from MUAs directly.

[JdeBP]

Indeed.

* http://jdebp.eu./FGA/qmail-myths-dispelled.html#MythAboutBar...