[arno1]

> This stack was created out of frustration due to the fact that to this day there's no easy way to have a full email server without the overhead of installing and configuring all servers needed to handle incoming and outgoing messages.

Interesting approach, though I solved this frustration by the use of a Docker and kept "my data is mine" + "no vendor lock-in" + "I control all the gears" approach. (Though, it's not perfect since VPS is ran by "someone" else.. but that place where you run this stack can be easily changed at your convenience). Simple docker-compose.yml with 3 images and voila.

This AWS S3 SES setup looks far more complex than what I did using only 3 docker images: the postfix (for smtp), dovecot (for imap), opendkim (for email sigining & verification). It's really easy to fire-up a VPS with a single click nowadays.

If someone is interested in the images I am using:

- https://git.nixaid.com/arno/postfix

- https://git.nixaid.com/arno/dovecot

- https://git.nixaid.com/arno/opendkim

Then you just feed the images with the right configs (main.cf, master.cf, .., dovecot.conf, opendkim.conf).

It's also possible to template the configs and make the variable-based configs. Make things scale friendly. I am also using Terraform to automate the server deployment/DNS record updates so it is easy to get from 0 to 100.

The only drawback is that you are the one to maintain the OS/SW upgrades, security, etc.. but that's something I really want to do by myself instead of relying on someone else :-)

[tuldia]

Thanks for sharing!

Your setup seems sane and pretty close to what would be if you want to run the same thing on a single server.

It makes the code base in the original link looks like a proprietary duct tape spaghetti :P

I just recommend setup postscreen[1] and rspamd[2].

1. http://www.postfix.org/POSTSCREEN_README.html

2. https://www.rspamd.com/

[fiddlerwoaroof]

Another “serverless” route for Docker containers is to deploy the containers with Fargate which isn’t too hard and gives you autoscaling without having to re-architect your application for serverless. (And has correspondingly less vendor lock-in)

[Canada]

> Interesting approach, though I solved this frustration by the use of a Docker and kept "my data is mine" + "no vendor lock-in" + "I control all the gears" approach.

Yeah, I totally understand the desire. And even hosting this on a cloud, you benefit from SMTP TLS sometimes, presuming no active MITM and the cloud service not actively abusing its privileges on your VM or storage. Which is probably not happening widely. At least as opposed to the protocol level logging that SES or similar services do for sure.

I recently opened some new accounts at AWS and other large one for a new company... first thing to do is setup mail of course. Both denied my request to allow SMTP. AWS ominously rejected me with some vague "maybe one or more of these reasons" including prior ToS violations or payment issues with "linked" accounts.

Frankly it's scary, and so far they are stonewalling me on any details as to what I "may" have done. AWS is one place I sure don't want to have a bad reputation with.

[brazzledazzle]

Fraud detection is such a frustrating double edged sword. They can’t share what was detected or why because the bad guys will start taking it into account. That leaves us with manual human review as the only means to address false positives. But that doesn’t scale so it’s either backlogged, low quality or nonexistent.

[m4rtink]

Rather seems likely laziness hiding behind the guise of "security". Such a bad user experience is inexcusable.

[fragmede]

It’s not a “guise” and not out of laziness.

It’s impossible to guarantee that the reason can’t get back to bad actors if you give that information out to anybody, so that information isn’t given out. If you can figure out a way, you’d have a bigger license to print money than Google and Amazon, combined. The problem is, rfc-3514 aside, there’s no evil “bit” and no way to tell if the person making a request is good or bad, or if they’re even the person who’s account they’re using. Don’t forget the possibility of an “inside job” either.

Sorry for the bad developer experience, but fighting all the various kinds of fraud is harder than it looks. Thankfully ML's made strides in this area.

[Canada]

I think the issue is that not very many customers want to maintain their own mail server anyway, so it's just easier to turn down these requests unless the customer asking spends enough money that you don't want to offend them.

[coder1001]

Do you mind writing a blog post somewhere explaining to noobs how this can be done?

This will be a great post with lots of traffic I imagine!

Thanks, great work!

[ab_testing]

If you are a noob with a domain and need to host your own mail server, you need mail-in-a-box. Look it up. It runs on a VPS and configures email and even a static site for your domain. It also comes with a dashboard where you can create additional email accounts.

[arno1]

I am actually thinking on writing it since long ago... But not motivated enough... :/

[angrais]

I'd read it! As would others, so perhaps knowing you have an interested audience might motivate you further?

[idclip]

You sort of almost already did with your reply. Just copy it, add pics, add code, add more text and details,

Done.

Do it.

[theSuda]

Please find some time to write it. It would be a great resource for self hosting.

[wishinghand]

Another vote for a blog post, please.

[social_quotient]

I’d read it

[jpdb]

> The only drawback is that you are the one to maintain the OS upgrades, security, etc

Since you are running everything in docker this could probably be made easier by using a slimmed down distro with automatic updates.

You'll still need to make sure your docker images are updated and patched though.

[arno1]

> You'll still need to make sure your docker images are updated and patched though

Of course. ;)

[wildduck]

Have you considered: https://haraka.github.io ?

I have being running it on a instance with 1 GB ram and it is running pretty smoothly.

[ascorbic]

Don't you have problems with deliverability? I've found sending emails directly from an AWS IP (i.e. not via SES) has major issues with reputation management. It's really easy to have outgoing emails mysteriously spam filtered.

[xmly]

I think his solution is serverless, no need to maintain the servers.

But both solutions are kind for developers, not for end-users.

[jijji]

complex but not to mention expensive. i wonder what the costs per month would be sending/receiving a relatively small workload per day (1000 messages) doing it with SES and S3 versus a cheap vps provider.

[social_quotient]

The SES and S3 side of things would be free?

(62,000 free per month) https://aws.amazon.com/ses/pricing/

Yes, complexity would be a factor for sure.

[why_only_15]

For receiving they only get 1,000 free, and for both after that it's $0.10/1,000 emails, which isn't very much at all.

[TheSpiciestDev]

> SES charges $0.09 per 1000 mail “chunks”, where a chunk is 256 Kb of data. This is on top of the base SES fee and S3 operation and storage fees.

> But it only charges for each complete chunk. So < 256KB is free, 256KB is $0.09/1000, etc.

Others elsewhere here have stated otherwise - haven't looked for the small print on AWS' page myself yet, though.

[uchiga]

what if you get 10 billion unwanted messages? 900k bill?

[sathyabhat]

It's 62,000 free only if the emails are being sent from an EC2 instance, else it is $0.10 for every 1,000 emails you send.

[danenania]

Another drawback is that while yes, you can scale up fairly easily with terraform, your server can also fall over if you get a heavy burst of traffic, and you'll return errors until you're able to provision more machines. Depending on what you're doing, how fast you're growing, and how much tolerance your users have for downtime, that might be a pretty big deal.

[vidarh]

You can set up autoscaling groups via terraform just fine, with a little bit of care taken to ensure that you trigger on the right metrics.

If anything mail is pretty much the easiest thing you can possibly pick to scale, because the inbound mail will be automatically retried. And haproxy in front of SMTP servers works just fine (really, any load balancer that can load balance raw TCP connections, but I've used haproxy to load balance thousands of messages a second).

For your user-facing side you need a bit more resiliency, but nothing stops you from using a service like SES to back a traditional setup for sending either. Reliably scaling outbound mail is the easy bit - the hard part is the reputation management that managed mail services provides, and no specific deployment mechanism will solve that.

[danenania]

Sure, but for heavy/bursty traffic, you can still have downtime while new VMs spin up. Retries might save you or they might make the problem worse, depending on the size and pattern of the burst and how your auto-scaling config interacts with the retry config of various hosts.

It may seem like a nitpick or something not worth worrying about, and for most that's probably the case. But for some businesses it could be a crucial difference. My point is simply that this is a legitimate benefit of serverless that wasn't mentioned above--I didn't think that would be a controversial point.

[vidarh]

That is no different for serverless. You don't magically escape startup times - you need to carefully ensure that cold startup times are low enough, or that you maintain excess capacity to compensate.

The precise extent is different between different platforms depending on overheads, but that just means the point at which you need to trigger scaling is different.

You can find lots of descriptions of approaches people have taken to keep serverless instances running to avoid the cold start delays to work around this... For autoscaling groups you'd instead configure the alarm points used to trigger the scaling accordingly.

Serverless platforms tends to assume the startup will be fast enough to keep the connection open rather than return an error, but that is a load balancer config issue - you can set up retries and wait for other platforms too if it makes sense.

(Though for email this really does not matter - retries for all commonly used mail servers follow some form of exponential backoff up to many hours at least; retries works just fine in practice)

[speedplane]

> That is no different for serverless. You don't magically escape startup times - you need to carefully ensure that cold startup times are low enough, or that you maintain excess capacity to compensate.

Serverless deployments are just another ladder step up the abstraction level, continuing the tradition that hardware doesn't matter. Similar to code compiled into assembly or a garbage collector managing memory. In the common cases, these cases are harmless (otherwise they wouldn't be popular), but they generally hide what's actually happening. Doing a garbage collection on a 200MB app is generally pretty snappy. But doing one on a 32GB server app can take seconds or minutes.

Abstractions like these are fine, as long as the limits of the abstraction are well understood. Sadly, that is rarely the case.

[crtlaltdel]

> Reliably scaling outbound mail is the easy bit - the hard part is the reputation management that managed mail services provides, and no specific deployment mechanism will solve that.

^this. if you want to send email its not hard....but if you want that mail to pass spam filters its a different problem altogether. hosted services like SES and mailgun will expose problems in how you are using emails (not handling bounces, not handling unsubscribes, etc) and in our case was very helpful.

[arno1]

Yeah, this is normal. One bus can't fit more people than it physically can.

The high load can be alleviated by the use of more MX server DNS records (and the MX servers of course, across the different locations), LBs, smarter thresholds. Of course nothing is a panacea.

Either way you will hit the AWS's limits or will get a huge bill. And then, even if you set up the budget limits, it still won't make the service more available once you reach the limits.

[danenania]

If you're running a saas and the increased traffic comes from paying customers, you likely prefer a huge bill to downtime.

But apart from that, there's a huge benefit in saying "I'm happy to spend any amount up to X" and not needing to do any capacity planning beyond that vs. continually trying to guess what's the right % to over-provision your VMs and having downtime when you get it wrong.

[arno1]

Yep, but how can you be sure the serverless provider will never go down? I've witnessed multiple times when AWS's services went down.

> If you're running a saas and the increased traffic comes from paying customers, you likely prefer a huge bill to downtime.

Well, in such situation, I would probably run more advanced container orchestrators such as Kubernetes which you will then configure to automatically spawn the additional server instances.

Of course there are certain advantages in running a serverless code as you have just mentioned, but since my primary concerns are "my data is mine" + "no vendor lock-in" + "I control all the gears", it is not the best option for me. Unless I want to run & provide the serverless services by & for myself.

It's always a game between the security (more freedom) and the convenience (less freedom). Though, for some, there is more freedom in the convenience (until they start to see their hands are in the digital cuffs :P)

[danenania]

The serverless provider can go down just like the VM provider can go down, but the key difference is that it won't go down due to traffic bursts.

Auto-scaling helps, but it still takes awhile to spin up new VMs, and you'll have downtime in the meantime for sufficiently large bursts.

On lock-in, in my experience with any non-trivial infrastructure you end up tied to your provider anyway. You're still going to need IAM users and VPCs and subnets and provider-specific auto-scaling and all that jazz. Serverless lock-in is deeper, but either way switching is a big project.

[arno1]

This is why the important projects must be thoroughly planned and tested in order avoid non-trivial infra or a blind cat situation. And then it is a really good idea to leverage IaaC (Infrastructure as a code) which would bring the whole thing up really quick.

[dillonmckay]

Email itself has built-in resilience for such scenarios.

[z3t4]

If it's a well behaving e-mail server it will keep trying to send the e-mail... A trick to stop spammers is to block all new connections for an hour. While spammers wont try again. Sadly some legitimate e-mail servers will not try again either :/ Also some e-mail servers wont try your backup e-mail server... Some servers will even give up if they haven't been able to establish a connection within a second. Some developers/admins give zero shit about edge cases and conditions outside their developer machine.

[kortilla]

It’s a mail server. Traffic bursts aren’t really a thing unless you’re servicing thousands of users.

[danenania]

Right, I'm looking at this more as a potential backend for an email-heavy saas, in which case I think handling bursts without downtime could be pretty important. If you just need a mail server for yourself or a small company, I agree it's not an issue.