[immibis]

It could, but why change what isn't broken? Email messages are quite agnostic to how you deliver them, though. IIRC Google and Microsoft use a proprietary protocol with each other. Outlook uses a proprietary protocol with Exchange.

Email used to be transferred through a non-internet packet-switched point-to-point network of intermittently connected dial-up links. The fact that the email messages are agnostic to how they're transmitted is actually very cool, because it enables switching to future technologies without any changes to the user agents, and vice versa. You put a file in your outbox and it magically appears in someone else's inbox later. Originally mail readers had to be executed on the mail server and read directly from your inbox directory, then we created new last-mile protocols so the reader could run on a separate computer (which in my understanding is basically identical to a Fidonet "point").

You can easily imagine a mail server connected to an onion router, or to Yggdrasil or some other meshnet, or even to whatever remnant of the original UUCP net someone is still running for fun. The current design enables this sort of thing.

[sangupta]

If these are basic questions, pardon my limited knowledge of email-transport internals.

- May be the transport is better than HTTP, but HTTP is well understood and easy to debug for developers (debugging a web-app is easy). Similarly, a JSON payload will bring structure than the current way of creating boundaries. Moving to HTTP+JSON will allow far easier access to developers who want to build on top of it, or self-hosting. Get a domain, run a web-app, set a few records and you are done.

- Put a file in outbox and it get's out. From my outlook experience, it seems to be a cron that scans a particular DB query. Should still be possible.

- I have no idea on how onion routers etc work so won't comment on that.

- And lastly, if Google <> MS, Outlook <> Exchange use a proprietary protocol then I will read that as an indication to improve current standard.

[immibis]

My vote for the most important improvement to SMTP would simply be pipelining. It was designed in a time of nodes with low memory connected by links that are fast relative to everything else, and therefore, it uses a turn-taking command/response paradigm where each side keeps having to stop and wait for the other. "Hello." "Hello." "user1@mydomain wants to send a mail." "Okay, continue." "He is sending it to user2@yourdomain." "Okay, continue." "He is sending a copy to user3@yourdomain." "Okay, continue." "Here is the data." "Mail accepted."

(at least there isn't an extra step after the data to say "that was all")

A redesigned version would send all parameters at once, and then get a single success or fail response at the end. Perhaps one pause could be useful to validate the headers before the main body is sent, but only for large messages (e.g. with attachments). "Hello, user1@mydomain is sending to user2@yourdomain and user3@yourdomain. Here is the message. Bye." "Hello. Sorry, user2@yourdosmain is unknown. Bye."

An extension like this exists for NNTP, in RFC4644, since that really is a mesh topology instead of everyone-talks-to-everyone and some central links have extremely high traffic. It requires two round trips per message and processing of different messages can be interleaved while waiting for the reply. "Do you want message 1? Do you want message 2? Do you want message 3?" "I want message 1. I already have message 2." "Here is message 1. Do you want message 4? Do you want message 5?"

[MiscIdeaMaker99]

Apologies if you already know this, but pipelining has existing in SMTP for decades now. You can see if an SMTP server supports it when using EHLO (instead of HELO).

  Trying 74.125.202.26...
  Connected to smtp.google.com.
  Escape character is '^]'.
  220 mx.google.com ESMTP 8926c6da1cb9f-489374a766fsi10850940173.171 - gsmtp
  EHLO testing
  250-mx.google.com at your service, [47.227.77.52]
  250-SIZE 157286400
  250-8BITMIME
  250-STARTTLS
  250-ENHANCEDSTATUSCODES
  250-PIPELINING   <============= HERE
  250-CHUNKING
  250 SMTPUTF8

I believe this RFC 2920 is the original standard going back to 2000. I remember pipelining being a thing in Postfix about 20 years ago, at least.

[immibis]

I believe you still have to stop at certain points to check the responses. You might not want to send a message if one of the recipients is invalid, but only that command will return an error code, and the message sending will succeed.

You'd also need to prevent command injection. If the response code to DATA is an error, but you sent the message anyway, the whole message body will be interpreted as commands. Oops! The line ending bug (SMTP smuggling discovered early this year) was bad enough.

[immibis]

... not sure why I decided to write the other comment here as it doesn't really address anything you said.

What value does HTTP add? Probably nothing, but it adds a lot more complication. You may think it's well-understood if you only know the happy path (requests.get("http://blah/") and magically get a response). Do you know about HTTP Request Smuggling, for example? Nobody knew about that for 20+ years. HTTP brings a lot of dark corners which are not well-understood.

The workflow with hypothetical Simple Mail Transfer Protocol (wait, that name is already taken... damn it): 1. Connect to recipient's server; 2. Send the mail as a sequence of bytes; 3. Receive response code; 4. Disconnect.

And the workflow with hypothetical HTTP Mail Transfer Protocol: 1. Connect to recipient's server. 2. Send the magic text string "POST /inbox 1.1\r\nHost: yourserver.com\r\nContent-Type: message/rfc822\r\n\r\n". 3. Send the mail as a sequence of bytes. 4. Receive the magic string "HTTP/1.1 ". 5. Receive response code. 6. Receive some more bytes you don't care about. 7. Disconnect.

I'm just not clear on what benefit is added by steps 2, 4 and 6. It's not the host header, since emails include their full destination address. It's not the content-type, since you already knew it was an email message.

---

The same, but to a lesser extent, with JSON. I don't think JSON has many vulnerabilities in dark corners, but it does have inconsistent implementations. Trailing commas? Comments?

If you're using JSON to just send the email then it's the same question as HTTP: what is added by the extra steps? If you're also converting the email itself to a JSON-based format (as opposed to MIME which is currently used) that might not be worse or better (MIME is pretty annoying) but it's also incompatible for no real reason. It will be a separate protocol, not an "upgrade" to email - may as well use Matrix instead.

It brings its own problems, some of which MIME also has. Because you can write things in any order in JSON, you could send the whole 800-megabyte attachment before you tell the server who the email is going to, and the server might have to buffer that in RAM or in a temporary file. In MIME, the headers always come before the body. JSON can have deep nesting, and a naive parsing of 9999999 open brackets followed by 9999999 close brackets will lead to a stack overflow. If you reject those brackets, now you're rejecting valid emails just because you happen to not like them. This problem also exists in MIME, but at least MIME nesting more than a few levels is extremely rare since only entire messages and files get nested (e.g. attachments, and plaintext versions of HTML emails), not metadata, while JSON also has nesting for metadata. Additionally, servers don't actually have to process MIME nesting in many cases, but can treat the whole message as one big byte stream. It's mainly only clients who have to parse them. Servers would have to parse the whole JSON message in case there's important metadata after it.

Also since the order can be anything, servers couldn't look at the recipient first, open a file in their mailbox, then write the mail to the file as it arrives, since they might not know who the recipient is until after the whole message has arrived.

Finally, JSON handles binary attachments even worse than MIME does.

---

You mention "get a domain, run a web-app, set a few records and you are done". Why do you believe that is something exclusive to HTTP? Get a domain, run a mail-app, set a few records and you are done, too.

The difficulty with setting up your own mail is with outgoing mail. Incoming mail is generally unrestricted. But when you want to send mail, the recipient's mail server has to be convinced it isn't spam, and that's really hard no matter what the protocol is. Whatever you can do, spammers can do it too. The Fediverse works a bit like you describe, with JSON-based mails going over HTTP, and just hasn't gotten flooded with spam yet because it hasn't. Also spam is a little bit out of fashion thanks to the good filters at places like Gmail. It used to be that literally 99% of email's in everyone's inbox would be spam - only 1 out of every 100 a real email - but that's no longer the case. I imagine that setting up a new spam ring isn't really worth it, but some spammers who set them up long ago just keep operating them since it costs little. It will happen to the Fediverse too; it has almost no spam defences.

A lot of hosting providers block you from sending emails (connecting to other people's port 25). That's because hacked servers that send spam are much more common than servers that want to send legitimate mail. Most of them will unblock it if you ask nicely. This mechanism doesn't exist for HTTP, but I'm not sure if this mechanism is required to stop spam these days anyway.

---

Onion routers means Tor. It has its own address space. The point is, it's a network that's not the internet. It's an overlay network, so you need an internet connection and specific software to use it. If there's ever a serious .onion mail infrastructure, you might get mail at your internet address from a .onion address via some kind of gateway server. I'm on foo.onion, my server (being set up as a special .onion mail server) knows internet mail can go to bar.onion, which is also bar.com on the internet, and it will be passed on to the right internet server at yourdomain.com. The fact that this is even allowed at all is something I find elegant about email (and Usenet) and it would be a shame to get rid of it for... basically no reason. It would still be possible to do this if it was all HTTP-based; foo.onion would connect to bar.onion using HTTP instead of SMTP and that wouldn't really make any difference. Scenarios like this one only become problematic if the new design requires two-way communication between the sender and the receiver, like a key exchange. They don't become problematic just because you added steps 2, 4 and 6 to the protocol.

A related scenario is networks with limited connectivity. There are proposals for wireless ad-hoc store-and-forward networks, with very high latency, where two nodes would exchange messages whenever they happen to be within range of each other - imagine people with cellphones walking around a city, and only a small percentage of people have the app. Whenever two people with the app sit in the same café, they automatically swap messages to get the messages closer to their destinations on average. The one-way latency could up to days, if the messages arrive at all. If they often don't arrive, then the network is broken, non-viable, so let's assume they usually arrive after some hours to days. Email can work on these networks because it's one-way. You put a message into the network, and it somehow (according to the network design) spreads around until eventually the recipient has a copy and can read it. The recipient can't send a message back to negotiate a secret key, because that would take more hours or days, and then more hours or days while the key negotiation result comes back, and so on.

Most protocols can't work in this kind of network. Email (and Usenet) can. It would be a shame to ever delete this ability just because of... not much reason at all. Protocols in the Internet age are designed for systems to be able to "directly" communicate with each other with up to a few seconds of delay. I put "directly" in quotes because there are actually many IP routers in between them. Email actually pre-dates the Internet, so it doesn't rely on the Internet existing. Actually, the environment which email (as well as other systems like Fidonet and Usenet) were designed for wasn't too different from the wireless mesh scenario, although with a more predictable schedule. (Actually, I think Usenet was designed for more expensive permanent connections, but it still has the same design)

Of course, corporations won't be sending you invoices over any weird networks. Nor do any major mail servers know how to find a gateway for an overlay network. In fact 99.99% of the internet won't be able to send you mail in these weird situations. So feel free to delete it from the protocol if you like.. .nobody will notice. I think it's a pretty cool ability though. When Earth and Mars establish communication and there's a 2-to-6-hour light-speed delay, people will definitely notice if the email protocol takes extra round-trips.

---

Several months ago I signed up for a Usenet account just to see how it works. The most interesting thing I noticed, other than the one-way store-and-forward synchronize-once-a-day-if-you-want design, is that it has all the exact same problems as modern systems, and the exact same solutions, and even the exact same caveats to those solutions, 30 to 40 years earlier than our modern systems. We have been reinventing the wheel all this time and haven't gotten anywhere. Well, I lied - the problems and solutions are a little bit different, but really not all that different.

I also noticed that ActivityPub is basically just email if you change SMTP to HTTP and change MIME to JSON. It might be exactly the protocol you're looking for, and the fact it's associated with federated Twitter clones is accidental - just like the fact SMTP is associated with email, since SMTP can send any blocks of text data, and ActivityPub can send any blocks of JSON data. You might be interested in experimenting with it.