Hacker News new | ask | show | jobs
by kbenson 1804 days ago
SMTP used to be a lot more reliable than it is now. Now, with all the changes to help with blocking spam, you have to be very careful or have a lot of control over the receiving server to ensure you actually get delivery. Some anti-spam systems will just discard if the matching rules indicate the spam likelihood score is above a certain threshold, and mistakes in rules at system levels can and do happen.

But here's another way you could (ab)use the mail system for delivery, provide a mailbox for the client and just allow IMAP or POP access and throw the messages into that. The client can log in to access and process them (which they would likely be automating on their own mailboxes anyway). It does mean it's housed at the provider, but it's also pretty easy to scale. There's lots of info on how to set up load balanced dovecot clusters out there, and even specialized middleware modes (dovecot director) to make it work better so you can scale it to very very large systems.

4 comments

I don't think mr throwaway was advocating to use email to send the events, only to use SMTP. Email is an entire ecosystem, SMTP is only a protocol.

If the distinction is too hard to make: think of it as using the 'Simple Event Transfer Protocol' that just happens to use exactly the same protocol as SMTP.

> Email is an entire ecosystem, SMTP is only a protocol.

Yeah, but it's a protocol for transferring email. As I noted with "you have to be very careful or have a lot of control over the receiving server to ensure you actually get delivery", you can amstract most of the mail system out as long as you ensure you are running the server they deliver to, but you would also need to rely on them making sure their outgoing server is good for this, which probably means dedicating it to this and not running any real mail through it (so you avoid outgoing company email filters, etc). At that point, both sides are running specific bespoke mail servers, which cuts down on the usefulness of the solution because of how much setup and administration it requires.

It used to be nobody ran incoming and outgoing filtering on email, so it was a robust channel for communication with retries, and notifications for failed delivery, etc. These days it's not exactly that because of all the spam mitigations and company compliance and risk mitigations that might be in place, etc. In fact, just setting up a new mail server and attempting to send to microsoft (live/hotmail), yahoo or gmail is extremely hard, because they have a high bar for acceptance, and large swaths of the easily obtained IP space have already been blacklisted from prior spam use so you start at a bad reputation and have to work to get it to a level you've even be allowed to talk to other by working with all the third party (and first party) blacklist maintaining entities.

It's not uncommon to set up daemons that only talk to each other for infrastructure monitoring and reporting.
Yeah it does make more sense to have the IMAP/POP setup rather than actively sending out emails through consumer level email services like gmail etc where deliverability might become a concern.
at that point you'd better off using an atom/rss feed
The difference is that using a mail subsystem to handle this handles a lot more of the implementation than "use an atom/rss feed".

Notably, in choosing to use an atom/rss feed, you need to determine what the webserver serving it is, how to implement authentication on top of it (is it a token/oauth, HTTP auth, param auth, etc), what is the underlying data store (SQL/NoSQL, some message system), how to scale that system if you expect it to be large and span multiple servers and/or datastores (mail systems right now deal with hundreds of thousands of users and gigabyte plus mailboxes of millions of messages).

Choosing IMAP to deliver this info means there there are well worn solutions for all the decisions you need to make (including howtos to implement oauth at the server level), as well as client level libraries in almost every language. Basically, you could decide to use it and not have to worry about forging a new path on that system basically ever, because there's plenty of people that have already implemented it at a larger level and with the same features (even if you would be using them to slightly different effect), and they've contributed the info on how to do it and what the performance ramifications are to the public domain.

I'm not seriously advocating for it, but that's more because clients will look at you funny than for any technical reason. Technically, it actually has a lot going for it. Unfortunately as an industry we fetishize the new and bespoke because obviously our own unicorn projects are so new and special and will serve so many people that some off the shelf solution could never be as good....