[davideous]

The freedom of choice is actually in the hands of the end-users.

The end-users choose which email providers (Yahoo, Gmail, Hotmail) they want to use. The email providers are motivated to provide a good user experience, which includes blocking unwanted email because they make their money through user engagement.

The key to delivering into the inbox is sending mail that your recipients both want and expect. Provide a good user experience, and you'll build a good reputation. Push the limits (for example, use a "pre checked" checkbox on an order confirmation page to put people on your sales mailing list) and you'll be putting out email delivery fires all of the time.

Most of the things that you describe (SPF, SenderID, DKIM, PTR, MX) are all technical requirements, which are just the baseline for delivery. These are required, but any sender of unsolicited email can configure them. They don't earn you access to the inbox. Just like properly formatted HTML does not earn you great SEO results.

I like to break email delivery down into four areas:

* who -- send to people who requested your email and are expecting it

* what -- send something of value to these people

* technical foundation -- (SPF, SenderID, DKIM, Feedback Loops, etc.) required, but having it does not give you any points

* monitoring -- (open ratio, complaint ratio, ISP response codes) you need to know when something goes wrong

(My company, www.drh.net, has been providing email server software, services, and deliverability consulting for over 10 years.)

---

[edit; added the below]

Another way the freedom-of-choice is in the hands of the end users is this: the big ISPs (yahoo, gmail, hotmail) make most of their filtering decisions based off of end-user behavioral data.

For example:

* what percentage of your email is opened

* what percentage of your email is complained about (the "this is spam" button)

* what percentage of your email is deleted without reading

* how long is your email read

* how much is forwarded

* how much is replied to

* what percentage of your email that was placed in the Spam folder when seen by the user received a click on the "this is not Spam" button.

This is the end-users voting on if they want your email or not. This isn't the entire email deliverability equation, but it's a huge part of it.

The ISPs treat this data so importantly because: (a) it's hard to game unlike content filtering, and (b) it directly correlates to good user experience which they want to provide.

[edit to make bulleted list look right]

[jt2190]

  > Provide a good user experience, and you'll build a good
  > reputation. Push the limits (for example, use a "pre 
  > checked" checkbox on an order confirmation page to put 
  > people on your sales mailing list) and you'll be putting 
  > out email delivery fires all of the time.

Believe it or not, it's possible to be a good actor, follow all of the rules and best practices, and still get flagged as spam.

Mail recipients are not perfect. They forget that they signed up for things. They accidentally click the "spam" flag on their messages. They get lazy and instead of unsubscribing they click the spam flag.

The real culprit here is that email messages rely on blacklists and not whitelists, i.e. recipients are required to give all senders full access and then block them when they misbehave, instead of giving them no access and giving them more access as they build trust.

So: What would it take to implement email whitelists across the industry?

[davideous]

Yes, good senders will still get messages flagged as spam. But the ISPs know this and they look at the complaint ratio. A complaint ratio of 0.5% or 1.0% is considered good. A complaint ratio of 3.0% is a problem.

We have one customer that's cleanest-of-the-clean (confirmed opt-in, valuable content, solid brand) sending 600k emails/day, and we see hundreds of spam reports. But their email gets delivered to the Inbox.

If you're a good actor with a solid technical setup you're still going to have an occasional delivery problem. This is why monitoring is so crucial. But you're not going to be putting out fires left-and-right, which is what it sounded like what _asciiker_ was saying he is doing.

[jt2190]

True, but _asciiker_ was also saying this:

  > Conclusion: There is no common standard because every 
  > major ISP can set their own standards. This will 
  > eventually force everyone to use the same services 
  > worldwide.
  > 
  > Where's the freedom of choice [of ESP] here?

My point being that the current blacklist-based system is broken from a "freedom of choice" perspective. The current system favors the established ESPs, as the cost of doing it yourself gets larger and larger.

[x0x0]

davideos' point, though:

email users want and expect

[_asciiker_]

Thank you for this insight, I agree with almost all of it.

As an e-mail services provider, I cannot or should inspect what my customers are sending. I can suspend them due to complaints of abuse but the damage is already done.

Same goes for tracking. I still say, block domain names, not IPs..

[davideous]

> As an e-mail services provider, I cannot or should inspect what my customers are sending. I can suspend them due to complaints of abuse but the damage is already done.

As an ESP, since you are letting customers send through your IP space, then a bad-apple can hurt the delivery of your other clients.

This is one of the big jobs that an ESP has. MailChimp, for example, has invested a ton of effort into detecting bad-apples as early as possible. (There are some really neat big-data techniques.) This is also why SES requires that you start with a smaller quota and build-up.

Some techniques:

* manually reviewing new clients before they send

* giving a new client a limited sending quota, so they build reputation with you over some time

* detect clients/campaigns with high complaints, high bounces, or low opens and take compliance action

* detect a partially-sent campaign with a high bounce rate and suspend it

* don't give any client an unlimited sending quota, so they can't hurt you too badly

> I still say, block domain names, not IPs..

There's a minimum amount of mail volume required to build a reputation. Many of your clients might not have this so they benefit from being lumped-in on an IP reputation.

I don't think IP blocking will ever go away, as it's an effective technique. The threat of an IP block also places some reasonable pressure on ESPs to police their client base.

[drcoopster]

I'm not sure you understand how IP blocking works and why it would be impossible to block domain names the same way.

[_asciiker_]

EDIT: "I'm not sure you understand how IP blocking works and why it would be impossible to block domain names the same way."

If you mean that on the same server the e-mail IP and server hostname will be the same for all the domains, then yes I understand.

But filtering can be changed to achieve domain level validation, not IP. It is done already on our servers.