[mr_blobs]

I setup my own mail server last year and shortly gave up on the idea. Setting the server up isn't the problem. The problem is keeping your IP off of all the major blacklists.

I wasn't even sending out mass emails and 30%+ of my email would never be delivered. I had to constantly check to see if my IP addresses were on the various spam lists (and fight to get my IPs off) and I just got tired of it.

Companies like Google have entrenched themselves in many things like email and are slowly becoming the only option out there. A large amount of email addresses are @gmail.com or run through one of their servers and they ultimately control whether the recipient receives/sees your email.

The 'promotions' tab in gmail also made things worse for many small businesses. Google doesn't want you competing for their advertising space and pushes any emails it deems a 'promotion' off to the side, so users don't actually see it. I'm not even talking about actual spam emails here, but emails users knowingly signup for and are expecting.

Many people don't realize just how much a handful of companies controls the Internet and your ability to make a living online.

[eikenberry]

Once you have SPF/DKIM in place and make sure your IP isn't already on a blacklist for some past (previous users) infractions you should be good to go. I've run my own mail server for years and have only had to remove it from a blacklist once. So once you get past some initial blacklist monitoring work, you are good to go.

That does bring up the point of how to do blacklist monitoring. There are various commercial services out there that will allow you to check for free and monitor 1 host or something (eg. https://mxtoolbox.com/). I'd prefer to run my own though, does anyone know of a good setup for this?

[SwellJoe]

Yep. I work on software that runs on thousands of mail servers (I work on Virtualmin), and it is entirely possible to run your own mail server with good delivery rates. It's honestly not even that hard, if you get SPF, DKIM, PTR records in DNS right, and you correctly handle bounces and unsubscribes promptly. We've been sending out a few thousand emails a day for over a decade without incident, and we have several people who run bigger mailing lists than we do.

Though there are a few minor caveats to this. Microsoft (Hotmail, Outlook, Live, etc. addresses) mail servers are ornery, in that they hold grudges against IP addresses for a long time (seemingly forever, as the server we moved to recently had been in our possession for non-email use for a couple of years, and it was still on a Microsoft blacklist from a prior owner's abuse), and they make you jump through a few hoops to get it removed. Even with SPF and DKIM, they rejected 100% of our mail until we got off of their blacklist. Our previous server never had that problem...but we'd been on the same IP for like five or six years.

You need to be on an IP that is dedicated and that you're going to own for a long time, and not part of consumer IP blocks; you can't effectively run a mail server on a cable or DSL line, even business class, without jumping through a lot of hoops. But, if you're in a colo, you'll be fine. This also applies to AWS and other cloud server IP addresses; as I understand it, huge swaths of them have been burned by spammers who spin up and spam until they get shut down, and then move to another.

So, I guess it's relatively tricky to get things working at the beginning and you may have to fight a little with some of the big email vendors, but it's not really an ongoing thing, in my experience. Get it right, and then don't spam or let your users spam, respond appropriately when abuse does happen, and you can run your own mail server relatively painlessly.

[eikenberry]

Not something I can run, but http://www.mailradar.com/ seems like a better free service. They let you monitor 5 IPs for inclusion on blacklists which works better for a personal setup where you might have more than 1 domain but wouldn't have more than a couple VMs as the MX servers.

[balladeer]

I signed up for mailradar.com and the confirmation email landed in the Spam folder of my Gmail account.

[yrro]

> First name is invalid (use min 4 characters)

But it's my name!

[jrnichols]

"Once you have SPF/DKIM in place and make sure your IP isn't already on a blacklist for some past "

you'd think that it would be this simple, but it's not. I have had SPF/DKIM set up from day one, a totally clean IP, doesn't show up on any block lists at all, yet i'm still having some problems delivering to certain ISPs. Verizon is the biggest problem right now. I had problems delivering to Gmail because my server didn't have a good enough reputation. Everything from my IP was going right into the Gmail spam folder and there was nothing that I could do about it except sit back and wait. It took weeks for Gmail to finally decide that my reputation was good enough.

It's scary just how much power Google has over many things email these days.

Now I have to convince Verizon that my IP is not a dynamically assigned one.

[bigiain]

Question: where's your IP address come from? I'd have guessed that common cloud VM IP pools are likely all trashed permanently already? I somehow doubt AWS or DO or Linode or Rackspace et al are worthwhile places to host an outbound mail server? I'd also guess ISP pools of home IP addresses are probably just as poisoned. Is proper SPF/DKIM setup "enough" to overcome that? (Or are my suspicions about pools of IP addresses unfounded?)

[eikenberry]

Cloud providers are not as bad as you'd think. I've tested several DO IPs using various checkers, as I've been thinking about moving my server there (currently at prgmr.com), and they have all been clean. SPF and not being an open relay seem to be the 2 important things to keep you off the blacklists. I still don't have DKIM (been on my TODO for a while... but lazy) and haven't been put on a blacklist in many years.

[startling]

I used Digital Ocean for a year. No problems, never sent spam. Then my mail started being spamfiltered, apparently because a neighbor was spamming. Seriously don't recommend using cloud hosts if you care about people receiving your email.

[bigiain]

Hmmm, thanks for that. I just checked my CloudAtCost VM, which I'd never have considered running outbound mail from, and it's not on _any_ of the 90+ blacklists mxtools checks. This astounds me!

[icebraining]

FWIW, I stopped using my home connection because it was listed on Spamhaus' PBL, you might want to check that out if you plan on using yours: https://www.spamhaus.org/pbl/

I'm using DO currently and it's been working fine, though it's just for personal email.

[kalleboo]

I'm surprised to hear there are home ISPs that still allow outgoing traffic on port 25... I used to run an email server at home, and both ISPs I used required you to route all outgoing email through their SMTP server (which presumably had an outgoing spam filter on it). This worked fine for me because it meant my outgoing mail had a good reputation.

[garaetjjte]

Why? I expect from ISPs to deliver IP packets to/from my address, without filtering on basis what is in payload.

Though once when device connected to my wifi got infected and started sending spam, I got angry (not e)mail from ISP, so I drop tcp/25 on my router firewall.

[kalleboo]

So you, someone technical enough to set up their own email server, was spamming people, and it presumably took days or weeks for someone to report you, and you to check your mailbox and get around configuring your firewall.

Now imagine the typical user who has no idea what the letter means or how to configure their router and just ignores it...

I'm surprised your whole ISPs dynamic IP pool isn't already on every spam block list.

edit: just realized you aren't the poster I was replying to, so presumably you're not running your own email server

[startling]

I used DO for a year without a problem, and then my IP was blacklisted (apparently a neighbor was spamming) and I couldn't do anything about it. Be warned, and frequently check if Google accounts receive mail frequently.

[wjd2030]

I ended up using an smtp service from Mailjet to get around this issue. 600 free emails a month.

[josephb]

How does your setup interface with Mailjet?

[Sir_Cmpwn]

FWIW you can appeal to most major blocklists to have your IP address cleaned. I had no problem doing this for my own mail server.

[TheGrumpyBrit]

I use DO without any problems. You need to make sure your IP isn't blacklisted before you start (and if it is, trash the instance and try again), and keep an eye on blacklists in case your range gets caught, but you should really be doing that anyway.

[brightball]

Don't forget DMARC. Setting it up strictly and monitoring it via something like dmarcian or other tools should keep your domain clean as long as the IP is safe.

[ProxCoques]

While I hear you on Gmail's overbearing influence, I'm not a sysadmin (I'm a designer and I spend most of my time talking to people about "user journeys" and fonts), but I run my own (and my family's) email using Ubuntu on a VM. There are lots of easy HOWTOS on setting up Postfix with DKIM and bind with SPF. I've got an alert on one of those blacklist monitors and not been in any blocking lists at all in several years.

So if I can do it, you can too.

BTW I use Gmail as the front end though - it picks up the mail up from my POP3 server and sends it out via my server over TLS/SSL. That last step was a bit complicated to set up since Gmail doesn't accept self-signed certificates for SMTP sending. But I managed to work out how to set up and renew a free cert from StartCom.

[abverma]

You might want to switch to Lets Encrypt or a paid SSL certificate provider.

https://en.wikipedia.org/wiki/StartCom#Criticism

https://blog.mozilla.org/security/2016/10/24/distrusting-new...

https://wiki.mozilla.org/CA:WoSign_Issues

[ProxCoques]

LetsEncrypt needs port 80 open to the outside so that it can verify/renew certificates. I don't have port 80 open on the mail server though.

[Whitestrake]

LetsEncrypt actually has multiple options available for validation. Only one challenge type, http-01, requires port 80 to be open. Another, tls-sni-01, requires port 443. dns-01 requires configuration of your DNS provider. I personally make use of tls-sni-01 and dns-01 in different situations.

[benevol]

> The problem is keeping your IP off of all the major blacklists.

I run several email servers, and this is NOT the problem. You just need to configure things correctly. It's not 1996 anymore, you need to set up things like DKIM, SPF, etc. Also make sure your hosting provider has not sold you an IP that was previously used by a spammer. [0]

That's all there is to your "problem".

Besides that: Your solution to Google's insane domination is to put even more of your life on their servers? When we know thanks to Snowden that Google is part of the NSA's PRISM mass surveillance program? Surrendering is not what winners do.

[0] Simply run a Blacklist Check here: http://mxtoolbox.com/supertool.aspx

[SturgeonsLaw]

I completely agree, especially with your "defeatism is doing your opponent's work for them" viewpoint, but if your goal is keeping information off the big provider's systems, then it's not gonna leave you with many people to email...

[awinter-py]

I think you're mischaracterizing the promotions tab. When users interact with a web merchant, they're exchanging money for goods/services -- they're not signing up for spam. Hiding marketing email is a feature, not an evil plot.

It's not just G that differentiates between transactional email (a receipt) and bulk/marketing mail. Trans/bulk are so different that email companies consider these separate products (e.g. mailchimp vs mandrill).

Generalizing based on myself, I can guarantee that people really hate receiving crap. At least I can visually distinguish the G ads from my email and ignore them.

[TheGrumpyBrit]

I found the Promotions tab to be a really handy "middle ground" between spam and actual mail. It usually contains mail from companies I've done business with, who I might be interested in receiving offers from, but I don't want those offers mixed in with my day-to-day stuff. Having them separately allows me to read them on my own time, and also lets me see at a glance if anyone is sending me too much. They're also usually the kind of companies who will honour an unsubscribe request.

[Veen]

> The 'promotions' tab in gmail also made things worse for many small businessesaid

Do you mean "made it harder for small businesses to get eyeballs on their spam"?

Most consider that a good thing. It's only marketers and spammers who regret the bundling of these "valuable messages".

[binaryanomaly]

That's why I've left gmail years ago. I do not want to support this kind of "business model".

I also do not want to invest the time to maintain my own email server. Imho there are good alternative solutions provided by companies such as protonmail.com or mailbox.org just to name two I have personally experience with.

As with everything it is mostly about overcoming your personal comfort zone and start to act. Good luck.

[mr_blobs]

It's not about being outside your 'comfort zone'. I've been running complex setups for years and am perfectly comfortable getting my hands dirty.

It's about the time and effort it takes to constantly fight the blacklists and the downtime for my business.

[zAy0LfpBZLC8mAC]

What are you doing to constantly have to fight the blacklists?!

I've been operating my own mailserver for over a decade and I haven't ever had to remove my IP addresses from any blacklists, and I haven't ever heard such a thing from other people I know who are operating their own mailservers either.

[viraptor]

There's still a middle ground available. You can move your emails to a smaller but serious commercial email provider you actually pay for. I moved to Zoho for example (I pay them directly and use own domain), but there's still loads of others available.

I don't get the complaint about the promotion tab though. I had the emails auto-labelled before more or less as private / adverts / notifications / mailing lists. This maps pretty well to Primary / Promotions / Updates / Forums. Anything that ends up in Social I just unsubscribe from. I'm pretty happy with their autoclassifiers. They don't make me not see emails.

[Kalium]

> The 'promotions' tab in gmail also made things worse for many small businesses. Google doesn't want you competing for their advertising space and pushes any emails it deems a 'promotion' off to the side, so users don't actually see it. I'm not even talking about actual spam emails here, but emails users knowingly signup for and are expecting.

The difference between what a business thinks a user has signed up for and is expecting and what an actual user is actually expecting in their actual mind tends to resemble night and day. For most users the "Promotions" tab was a godsend that rescued them from significant amounts of email that was swamping the stuff in their inboxes they expected and wanted to read. It allows users to engage with promotional material at their own choice and in their own time.

Google's smart filters have been a fantastic win for the user experience.

[Whitestrake]

Agreed, for this particular point it's less about small business vs. Google and more about small business vs. users. Google stepped in and saved a lot of users a lot of time and effort. I'm not sad that it's tougher for businesses to consume my time while I'm cleaning my inbox out, I can go over the promotional stuff when I actually want to.

[vacri]

My company's mail is on Gmail, and because of some missetting on google's side ('webmaster@' and 'abuse@' addresses not mailable), we ended up on the RFC-Clueless blacklist.

I got that fixed, and have submitted half-a-dozen requests to have us removed from rfc-clueless, and they've all been ignored (and do their best to hide how to submit and what for). Fuck rfc-clueless - it's a blacklist for 'people who don't follow the rules', but they're bad netizens themselves and don't follow their own rules. Just... fuck them.

I can't imagine what it's like to end up on one of these ignored blacklists when you don't have the might of a professional email service behind you.

[jwr]

I've been running my own mail server since 2001. Only once had a problem with blacklists when I moved the server to a new IP at a provider with bad spam reputation. Otherwise, this has never been a problem.

Don't let Google and Facebook monopolize our communications.

[cm2187]

> Don't let Google and Facebook monopolize our communications.

I do the same but the reality is that 99% of my correspondants use gmail, yahoo or hotmail. So these companies get a copy of all of my communications...

[ericd]

I set up my own mail server and was sending bulk email (opt-in) with relatively little experience. Set up dkim and spf, and you should be fine. A bit arcane, but it's not the maintenance nightmare you imply. Once you get into automated bounce tracking, etc, it's a bit trickier, but no need to do that for personal email.

[empath75]

The promotions tab is the only reason I even look at marketing email. Otherwise, I'd just ignore it completely. Every once in a while I skim through it and clear it out.

[jack9]

I've been running a postfix for years (since 2008ish) and never once checked or worried about blacklists. My 3 users haven't reported any problems at all.

[viraptor]

> never once checked or worried about blacklists. My 3 users haven't reported any problems at all.

What if they missed emails they never knew about in the first place? What if they tried to contact someone and just assumed they've been ignored? Unless you know you're not on blacklists, you should be worried at least a bit.

[cobbzilla]

If you're only concerned about delivery and can live with a third-party SMTP relay, I'd hope you could set up this system to relay through sendgrid or mailgun. If it's only for personal use, you'd probably be fine on their free plan indefinitely.

[_andromeda_]

Any thoughts on SES?

[cobbzilla]

I suppose any third-party email relay would give you better delivery guarantees than you achieve yourself, at least at the outset.