[KirinDave]

Well, part of this is on us. Let's think on the reasons Gmail is so popular:

1. It's very easy to get to.

2. It has incredibly fast search that has 0 setup.

We have never really even tried to address problem 1 as an open source community. Networks, name lookup, and VPNs remain incredibly complex topics that beginners cannot hope to wrestle with. The best we have is .mdns which either works magically or perversely refuses to work.

Similarly for free text search, the software world simply hasn't delivered a lego-like solution for email search. You CAN rig up any number of open source projects but it is neither easy nor instant. And even other professional products like Apple Mail struggle with a mere gigabyte of email.

Despite the fact that it's 2021 and every successful email provider aggressively solves these problems, the open source world still debates about the utility of ubiquitous search or pretends that local networking isn't a pressing problem.

[jerf]

"We have never really even tried to address problem 1 as an open source community. Networks, name lookup, and VPNs remain incredibly complex topics that beginners cannot hope to wrestle with."

I kinda disagree. It's probably easier than ever to set up your own mail server, in some abstract sense. You can get a virtual machine, use docker, heck, someone can hand you a complete image that you just have to bring up and set up with some config.

The problem is, it literally doesn't matter how much the 'open source' community comes together, it simply can not provide a turn key solution as good as

     Desired email account: [________]@gmail.com
     Password:              [________]
     Verify Password:       [________]
     [X] I agree to have all my data used in arbitrary ways

It's not possible. There is no way to set up a server that easily, even in principle.

Or at least, not in a sane way. I can set up a site where you feed me your credit card number and pick a domain name, and I set up your AWS account for you, register your DNS name for you, configure DNS, and stand up everything you need and set it all up... but then we've got a split ownership interest. I can hand it all back to you, but you don't understand the setup. I can give you root on the system, but when you change anything, my automation stops working.

[ryukafalz]

>It's not possible. There is no way to set up a server that easily, even in principle.

I partially agree, but I think we could get a lot closer than we are now. It feels like the main reason this isn't possible is because you need to go through a registrar to get a DNS name, and that's tricky to do as part of a FOSS project. Maybe you could integrate with the APIs of a few registrars, but... it's not ideal.

As far as the "run thing on server" side of it goes, though, projects like Sandstorm[0] have gotten really far re: making it a simple process. I stood up this instance of Etherpad with a few clicks on a web UI, for example: https://sandstorm.terracrypt.net/shared/aR2HXaoLSkLuXLhhAQon...

Sandstorm in particular doesn't quite work for mail servers just because the software is heavily oriented towards webapps, but there's no reason a similar system couldn't work in principle.

[0] https://sandstorm.io/

[jerf]

The involuntary back burner started designing "how would I set this up as a business", and I definitely ended up with setting up docker containers. Sandstorm, unfortunately, is an example of being a bit too ahead of its time... sandstorm has a lot of stuff in it that just isn't that big an advantage to me vs. being in a Docker container. (I'm aware of at least some of the differences, but... it's just not that helpful to me for the most part.)

I think something that could be like sandstorm but using docker containers might be a good thing, but there's still the "you need to hand me your credit card if you want me to register a domain for you" which is a big barrier vs. the screen I showed above. A credit card off the top is already 4-6 entry fields, and everyone knows to be a bit nervous about entering those....

[Avamander]

> It feels like the main reason this isn't possible is because you need to go through a registrar to get a DNS name

I can't agree, getting a domain is much easier than it is to configure the abysmal shit that is Postfix+Dovecot+WebGUI. God forbid you want proper search as well.

[ryukafalz]

Basically all of that could be done programmatically though. There are sane defaults that can be used for these things; projects like Mail-in-a-Box do most of that heavy lifting for you. (When all you want is a basic SMTP/IMAP/webmail server... a standard config can work just fine.) It doesn't even necessarily have to be done through a command line.

What can't so easily be done programmatically is DNS registration, because it involves money changing hands and there's no standard "registrar API". You'd have to either support an arbitrary number of differently-shaped APIs of different registrars or pick a "blessed few" to support.

[tracker1]

Realistically, you could support half a dozen DNS providers and get a LONG way towards making it easier to roll your own.

GoDaddy, GCP/google=domains, AWS, Azure, DigitalOcean and a couple others would provide enough for a LOT of people. Especially since godaddy and google domains include dns, and DO doesn't charge for it.

I've setup a script to make it easier to manage a handful of dev domains (commit to a repo, and it updates via build/event pipeline) with DO.. was surprisingly easy. I don't think doing similar for the biggest cloud/dns providers would be all that hard to integrate.

[sumtechguy]

Another issue that needs to be solved is 'bad actor'. Basically someone gaming the system and overloading it with spam. There is no real nice neat way to fix that. Instead it is a mishmash of blacklists/whitelists/blocklists and sorta intelligent alg filtering. Getting that all filled and working is not trivial either. Oh its 'doable' but kind of a pain for even someone with decent experience at it.

[ywei3410]

To add to that; with re-assigned IP's, widespread port 25 blocking /most/ ISP connections won't allow you to run an email server on a residential connection.

Spam/botnets have a lot to answer for.

[creeble]

Agree 100% on both of these, and #2 is seldom listed as a big issue. Had IMAP addressed search in a better way, I think it would have made a huge difference.

Of course, it's not simple to set up a mail server that stays clean of spam RBLs, or that is Gmail-acceptable out the gate. But that's just the bar that got set as people went to Gmail because of the lack of other good alternatives.

Question is, is Gmail hegemony fixable now?

[thayne]

I think the best approach to breaking the Gmail hegemony is to replace email with something better. Some features I think such a system should have are:

1. End to end encryption 2. search as a first-class feature 3. decentralized/federated 4. standard minimal rich text format (maybe something like markdown), instead of an inconsistent subset of html 5. Fix some of the legacy limitations of email, like having to be ascii safe, line limit of 70 characters etc. 6. Possibly make it easier to have an identity that isn't tied to your service provider

The big questions are how would you get the general population to switch from email, and what to do about spam.

[aidenn0]

4 there's already a standardized rich text format for emails, called "enriched text" defined originally in rfc 1563 almost exactly 27 years ago, and last updated in rfc 1896 about 25 years ago.

It supports many things that markdown does not (setting font families, sizes, and color for one example), and nobody uses it anymore.

[thayne]

Another point is that markdown is still pretty readable as plain text, whereas html, or "enriched text" are a little harder for humans to read if it isn't specially formatted.

[thayne]

and afaik hardly any email clients support it. and even if yours does, chances are the recipient on the other end uses one that doesn't.

[aidenn0]

Right, so I don't know what inventing a new such format will do to improve things.

[anthk]

1> GPG.

2> Your MUA does that. Get one that doesn't suck.

3> Impossible and/or dangerous.

4> No. HTML is NOT part of a MIME standard, even.

5> That's an issue of netiquette, often you have to post lines over 70 chars in News servers, too.

6> That's a different issue.

[thayne]

1. GPG has a lot of problems. Perhaps the biggest of which is that there isn't a critical mass of people who use it, and it is pretty difficult to us at all, much less to use correctly.

2. I have tried several MUAs and haven't found one that IMHO "doesn't suck." Although search isn't always the problem. (The biggest problem I've run into is actually getting them to work with logging in using an SSO method like SAML or OIDC). And this point was specifically in response to the parents statment that "Had IMAP addressed search in a better way, I think it would have made a huge difference."

3. Not sure what you mean by that since email _is_ decentralized and federated. Meaning, if you use gmail as your email provider you can still send messages to people using a different email provider. As opposed to say something like Signal or Whatsapp, where you can only message people using the same service as you.

4. Well, some subset of html has become the de facto standard for rich text in emails. Although what subset that is depends on the client...

6. Different issue from what? It is a very real complaint I have with email. Changing your email address is a pretty big pain, possibly more so than changing your physical address or phone number. For many people, myself included, the thought of having to change my email address is a pretty good motivation for staying with the service I currently use. Now it is possible to use your own domain with many email providers. But doing so is beyond the technical skills of most people, and costs more for essentially insurance in case you stop liking your current email provider some time in the future. Unfortunately, I don't know of a good solution for this. At least not one that is scalable.

[secabeen]

Yeah, none of the IMAP server authors really thought of search as a first-class feature, nor did they seem to want to do the development to build the backend support. Resource constraints were also an issue back then, as well as a client-first approach. Most large mail servers only had enough resources to run the IMAP and POP services themselves; adding search index maintenance would have overloaded their CPUs. (Not that they couldn't have added more CPU, but this was all pre-cloud.)

[Avamander]

> Question is, is Gmail hegemony fixable now?

Maybe, if we get maddy working near-perfectly and kept up-to-date.

[teloli]

I do agree with you, but let's add:

3. dealing with spam is a massive PITA

[Avamander]

Most of spam problems could be solved if mail software came with sane defaults. Sane rate-limits, sane IP reputation, sane security rules, some backscatter protection. Something. Most setups are so widely open because good configuration is arcane knowledge.

[varjag]

I use offlineimap/mu4e over Exchange account at work, in parallel to a personal Gmail account. Would not say now that Gmail has a perceptible edge there.

Seriously, mu4e search on modern hw is incredibly fast (and I have a 10 year work email archive).

[michaelmrose]

What about mu plus mu4e

Thunderbird plus global search

Recoll + client of choice

https://www.monperrus.net/martin/recoll-maildir-thunderbird

[thayne]

3. A whole lot of free storage, so most people don't have to worry about needing to delete old emails.

[throwaway201103]

For the search problem, notmuch is pretty easy and has better search capabilities than gmail.

https://notmuchmail.org/