[alex_studer]

Maybe slightly unrelated to the main topic, but:

> If you’re worried about mail deliverability issues, don’t be — it’s more or less a myth in $CURRENTYEAR. If you set up DKIM properly and unlist your IP address from the DNSBLs (a simple process), then your mails will get through.

Does this match most people's experience? Or is he just talking about the deliverability of smaller providers like Migadu? I run my own mail server and can never send emails to people on Gmail/Microsoft 365/etc unless it's in reply to an email that I received.

I've checked DKIM and the DNSBLs, and everything seems correct...is it just a matter of not having enough volume for email providers to trust me?

[jacquesm]

No, it doesn't match my experience at all. Microsoft, Apple and Google all tend to randomly drop messages from addresses that were perfectly good the day before without any change in the sending mail servers configuration. It is super annoying because it actually forces me to use one of those to send email and that actually rewards those companies for creating this problem in the first place. I'd rather deal with spam than random delivery issues without any knowledge about it.

Oh, and I get more and more false positives in gmails spam folder as well, and since they're deleted after 30 days I now have one more 'inbox' to monitor.

[elric]

Yeah that's my experience as well. And it's impossible to find anyone to talk to at Google/MS/Apple about an email issue -- unless you send really large volumes, then suddenly all kinds of support and tools become available. IMO it's a deliberate attempt to get more people to switch to gmail. But maybe I'm a cynic.

[thu2111]

The problem is that providing email server debugging services is very expensive because not many people really understand the whole SMTP stack these days. You can't find anyone who will talk to you at these firms because there's probably only about 20 people in the company who could genuinely help. Additionally the world is filled with people who would like to run a whole email server for a stream of personal mail who would get it wrong and demand debugging services. That's why they spend their time building automated tools.

The problem the article discusses about needing to send lots of mail to qualify for these tools is because the systems don't scale infinitely - data storage still costs these firms money - and botnets mean basically every single IP on the internet will try to abuse your service at some point or another. So tracking deliverability data without any thresholding means tracking it for every connected machine in the world, which is hugely expensive, all to satisfy the tiny minority of people who feel passionately that they should run their own email server. The only possible justification for that would be recruiting related, but they don't have challenges recruiting.

[jacquesm]

The problem is the old cars. Really, old cars have absolutely no business on todays roads, and if you want to park your old decrepit car in my brand spanking new garage you should upgrade your car, better still, buy one of mine. Of course I can't be assed to make sure that your old car can still park in our garages. Your track width is ridiculous and the amount of horsepower you have just doesn't cut it today. So keep your silly little museum piece that's all of 10 years old now and drive around in circles on your own lot.

If you have a few million of those or are part of our circle of buddies we just might cut you some slack. But otherwise, no matter that your vehicle performed fine just yesterday, it's up to us to change the rules at will and demand that you adapt.

[thu2111]

You're ascribing bad faith where there is none. Spammers make it difficult to keep the email system alive at all, even between big firms. If you think it's easy, go set up your own webmail firm and show them how it's done.

[jacquesm]

Carjackers make it hard to keep the road system usable at all, even between large fleets of cars. If you think it is so easy, why don't you set up your own fleet management system and show them how it is done, you miserly driver.

[dunham]

Apple will silently drop an email (no bounce message) based on text content of attached pdf files. This happened at my day job, we found the page of the pdf that triggered it via divide and conquer.

[Cu3PO42]

You could pay for hosting from a smaller provider on your own domain. Sure it costs money, but you don't have to support those giants and you get to keep your own address.

[joshuaissac]

It seems they already do that, which is why their e-mails are getting dropped when sent to a recipient hosted by the giants.

[Cu3PO42]

I was reading it as they are self-hosting and are therefore having their emails dropped. While I have had similarly terrible experiences with self-hosted mail, I have found even smaller dedicated providers have absolutely no issues with getting mail delivered.

[loloquwowndueo]

+1 these days I need to wrangle two inboxes. Thanks google.

[upofadown]

>Microsoft, Apple and Google

Well what do you want for nothing? Randomly losing email is what the free email services do. That is hardly the fault of everyone else...

[fomine3]

Microsoft 365, G Suite

[throwanem]

Deliverability issues for personal servers are emphatically not a myth. One reason I moved off self-hosting mail a couple of months ago, after 17 uninterrupted years of doing so, was because of perennial problems with deliverability - and Gmail, while bad, wasn't even the worst; some large providers (glaring daggers at you, Microsoft) make it effectively impossible even to understand why mail is getting blackholed, much less do anything about it.

(Fastmail, in case anyone is wondering. $50 a year, and there's nothing I want that they don't do.)

[Cu3PO42]

It absolutely does not. I tried setting up mail on my own server and by all accounts I did everything right. Mail-Tester and similar services found no issues, I had all the optional records including DKIM, SPF, etc. and yet I couldn't rely on my mail going through.

Microsoft (Outlook.com) was particularly bad: mails were silently dropped without notice to me or the recipient even when I was replying to a mail they sent. I tried adding my e-mail address on my own servers to the trusted senders list on a MS account of my own and even then not all mail would go through.

Gmail ocasionally put me in spam but at least always delivered my messages.

At that point I gave up since I consider email to be too essential to be flaky and paid for Migadu. I have been very happy with their service and appreciate their pricing strategy (charging for number of mails sent, not the number of domains or mailboxes added). That said, I still dislike paying for hosting that I had sufficient capacity for on my own servers.

[kureikain]

It does but only for IPv4. If you have those:

- reverse ip. origin ip -> dns -> resolve to ip

- SPF

- DKIM

- DMARC

- Make sure ip are not on DNSBL

- Message ID

- Try to connect to gmail mx server using TLS

Then you have very low chance to get in spam unless you are sending spam and get blacklist. Time to time an gmail mx server will reject you, rate limiting, just try again with another and you will be ok.

Source: I run https://hanami.run and have to deal with this a lot due to nature of an email forwarding service that people usually use as one-off email ([e-mail] for their one-off service) so it attract a lot of spam which I have to filter and make sure it won't go to gmail/microsft

I would say gmail is the best among big provide: Microsoft/Apple(iclouds) are garbage mail server where they just blindly trust DNSBL and block you no matter what. Gmail was way better.

However, if you use IPv6, it's a bit harder to avoid being flag as spam. It's really random and I don't know why yet and I contacted them and now "Waiting for 2 weeks to get a response"

[alex_studer]

Out of curiosity, where do you send the emails from? (Are the IP addresses from AWS/some other large provider? Or something else?) I've heard that the IP address block has something to do with getting marked as spam.

I've checked everything on that list and it's all good, and I'm sending from IPv4. The really frustrating thing is that Google's own Postmaster Tools [0] doesn't want to tell me anything without "a sizable daily volume of email traffic (up to the order of hundreds)".

If it doesn't have to do with the origin IP's neighborhood, my only other guess is that I don't have enough email volume, which seems like a catch-22...

[0] https://gmail.com/postmaster/

[kureikain]

I used Hetzner.

At the time I allocate an IP address, it's was listed on DNSBL. I tried like 30times to get a good IP and gave up. I instead just allocate a new IP, wait a week. Started to send email to myself every 15minutes to a gmail address. Anytime it isn't showed up in Important, I go in to move it to important. If it showed up as spam, I marked it as not spam.

After 2-3 weeks doing that, The IP is no longer listed on any blacklist and I started to roll them out.

Don't worry too much about IP blacklist. They aren't super important to gmail. If an IP stopped sending spam, they will be removed from IP blacklist eventually. Give it a week.

> "a sizable daily volume of email traffic (up to the order of hundreds)".

I send about 8,000 emails per day and somehow that Postmaster Tools not reports anything at all. I would say

If you send me an email [e-mail] I can tell you what you can do to avoid being flagged at spam. Happy to jump in a 1-1 chat https://calendly.com/vinh-hanami/15min to help you on that.

[jacquesm]

Even replying to gmail messages or forwarding an older address to gmail will randomly fail. I've had some pretty important messages (for instance: invoices) randomly blackholed while all other mail arrived fine. It is super frustrating to answer someone's request for an invoice and to see that invoice then disappear without any notice that it was spam trapped.

[kureikain]

Yes, I heard you.

To gmail point of view, they don't care about forwarding or replying to an email someone send you at all. All they care seems is about the message follow standard best practice(SPF/DKIM/DMARC/PTR) and send from a good IP. Sometime your IP can be flagged as spam wrongly on DNSBL. But it's very easy to get remove from DNSBL. Just submit a removal request, explain your situation and it's usually remove within a few days.

Also, the reason while your important emails are marked as spam is depended on gmail mx server you connected to. I don't know the detail but I have seen gmail mx server rejected/spammed my email, but on subsequent try if I connect to different one(by resolving other MX record), the email went through just fine.

If you send me an email [e-mail] I can try to spot if anything may randomly flagged your email.

Also, did you have multiple SMTP servers?

[bluecalm]

Not at all my experience. I was sending paying customers emails with their license key and a link to our starting page for years and it was fine. Recently our emails randomly land in spam v even though:

-we have never sent one unsolicited email

-100% of those emails are opened and read (I know because customers are activating their licenses). It's hard to imagine anyone mark the email with license they paid several hundred dollars for as spam

-we get tons of engagement, we answer around 30-40 emails a day

Something is terribly wrong with Gmail spam filter recently. Some of the emails from my customers land in my spam folder as well. They more often than not don't contain any links or anything out of the ordinary. Just a simple question or two.

It started happening a few months ago. It's a huge pain. All Google support tells us is that their magical box they can't disclose thinks our emails are spam even though there is nothing suspicious about either our website, the link or the content of the email. They can't tell us what triggers the filter cause it will make the job of the spammers easier.

I am now left to play the game against spammers: they try to imitate legit emails, if they get close enough then my emails are similar to spam, I have to modify them in random way, then spammers catch up etc.

Yay for machine learning I guess. Doesn't work so well even you have an adversary trying to fool you.

Btw, we use Gmail and it's Gmail spam filter which is causing problems. Google analytics is able to guess our monthly revenue and number of licenses pretty well. We send exactly as many emails. We are paying customer of Google Suite. It's still not their problem that their magic box puts our emails in spam but it's all on us.

[anonydsfsfs]

> Does this match most people's experience?

It does not. I used to work for a non-profit that ran our own SMTP server, and we had issues with our subscriber e-mails getting marked as spam, mainly for Office 365 users. We set up DKIM, DMARC, and SPF, but it still happened. We ended up having to move to using Google's SMTP servers just so our e-mails would go through.

[syntheticnature]

As someone who runs a discussion e-list server for a small volunteer org, Gmail has not been an issue with a good config, generally. The other big providers are annoying, to varying degrees, often silently dropping email. ISP email tends to be a pain in the neck, putting one on a blacklist if you exchange email with two different addresses they provide service to within one hour, though at least they tend to be noisy about it.

[jcelerier]

> Does this match most people's experience? Or is he just talking about the deliverability of smaller providers like Migadu? I run my own mail server and can never send emails to people on Gmail/Microsoft 365/etc unless it's in reply to an email that I received.

I use a gmail address and all my friends's self hosted mail have always ended up in spam by default

[sildur]

It does match my experience. I have a personal mail server with every bell and whistle working and my emails arrive to gmail addresses. My IP address was used by a spammer in the past, and took me some time cleaning it up in a bunch of dnsbl lists. But it reliably works now.

[jnwatson]

Yeah, that's not been our experience, and this is just for password reset/logins. You pretty much have to use a dedicated provider for that.

[baobabKoodaa]

Delivering email from your own little server is theoretically possible, but extremely difficult to do in practice. I wrote about this at length here: https://www.attejuvonen.fi/dont-send-email-from-your-own-ser...

[ZWoz]

I have run small servers something like 18 years, my own personal something like 15-16 years. I still don't have much trouble. Only issue I had few years ago was spamhaus suddenly listing my IP as residental, that was annoying few hours.

[baobabKoodaa]

How did you measure your deliverability?

[ZWoz]

Well, for personal matters I don't write usually letters without needing feedback. When I get answer, then my previous mail is clearly arrived. My friend has forwarding to gmail address, so he can compare both ends. Over years there isn't been more issues than mentioned spamhaus incident.

[baobabKoodaa]

Ok, so you actually haven't measured your deliverability in any way. Yes, you can probably get emails delivered to your friends who have already emailed you in the past. Spam filtering is more of an issue when you try to email someone who has not engaged with you. If you actually go out and run some tests, you will probably find out a good portion of your email is blackholed by Google and Microsoft.

[ZWoz]

How to you really test that? Any provider is autonomous, you really don't verify delivery. You have few technical options, like DSN or MDNs, but servers or clients don't need honor them. I didn't mentioned clients, I had few of them too. Some used internal mailing list style things, group addresses forwarded some outside service, like hotmail or gmail. I didn't had any complaints from them. Now talking about personal experience, your assumption is wrong. I don't change mail only with my family or friends. When you write merchant or government agency there is even more initiative to notice missing answer.

[noja]

No it doesn't. My mail server is massively rate limited even a year later.

[nr2x]

yeah, that's a 100% bullshit claim.

[superluserdo]

It doesn't match my experience but in fairness I was running the server off my home IP. I think it was the lack of reverse DNS that was tripping it.

[ddevault]

Send me an email: [e-mail]

I'll see if I notice any configuration issues and report back to this thread.

[Nux]

There's a site for that: https://www.mail-tester.com/

[looperhacks]

Weird, for my server spam assassin tells me my dkim signature is invalid but the dkim check in the next category says that it's valid.

[alex_studer]

Same here!

[alex_studer]

Done, thanks!

[ddevault]

Copying my reply to HN:

Some feedback: you should add a PTR record (reverse DNS) for studer.dev and mail.studer.dev.

I also noted that your DMARC record has p=quarantine, which depending on the details of your outgoing setup, may cause your emails to be placed in the spam folder if DMARC fails for any reason. I generally advise against using DMARC with any policy other than `v=DMARC1; p=none` or else you're liable to run into issues, it's not a very good standard.

Otherwise, your configuration looks good to me. Also try https://www.mail-tester.com to see what they think.

[alex_studer]

In case anyone is curious: I checked and the PTR record for my mail server (168.62.161.121 -> mail.studer.dev) is valid. mail-tester.com also gives me a 9.9/10, but I just tried sending an email to a Gmail account and it went straight to spam :(

I suspect it's probably a volume thing? As in I'm not sending enough email for Gmail to make any judgements about my mail server, since I'm only an individual. Of course there's not really a great way to resolve that...

[smartbit]

For gmail, try adding an IPv6 address with proper reverse resolving. Tip from HN some time ago.

[joana035]

Yes, it works for me.

SPF, DKIM and proper DNS records and that is it.

[shock]

Is your mail server IPv6 enabled?

[smartbit]

IPv6 magically makes email go to the inbox of gmail. A tip I read here in HN some time ago.

[WGH_]

It really depends. In our case, Gmail rejects our IPv6, but accepts IPv4. The IPs are from university network and they don't have any history other than being used by us. SPF, DKIM and PTR records are in place.

> Our system has detected that this message is likely suspicious due to the very low reputation of sending domain. To best protect our users from spam, the message has been blocked. Please visit https://support.google.com/mail/answer/188131 for more information.

This message is extra confusing because mail from said "sending domain" is accepted by IPv4.