[gettingreadyhn]

these articles push me more and more to drop using gmail in favor of "self hosted email". i really do not understand why we do not think and give it all up for free to google (and compromise our personal security in the mean time).

[derefr]

* Gmail's search and spam-filtering are both very good, trained and tuned on datasets no self-hosted product could ever match (and harnessing parallel algorithms across large clusters that'd be quite costly on one machine)

* Google doesn't lose my email; I probably will lose my email, because an email server is backed by a database and doing database backups right is hard if that is not your day-job.

* You can get a good email-receiving experience, but email-sending is very difficult these days if you're a nobody, because a lot of first-stage network-level spam filtering has come down to reputation, and your server IP won't have any (or, if it's a cloud provider IP, will have very likely been used at least once to send spam in the past.) And residential ranges get dinged, too, from the heuristic (stereotype) that the most likely reason to get an SMTP connection from a residential IP is that it's a member of a botnet.

[megous]

> Gmail's search and spam-filtering are both very good, trained and tuned on datasets no self-hosted product could ever match (and harnessing parallel algorithms across large clusters that'd be quite costly on one machine)

As someone who self hosts, this is clearly not true. With gmail I was receiving a lot of spam from various email marketing companies like mailchimp, easymail, etc. There's a lot of these companies and they are mostly country specific, some less, some more shady.

With self hosting it is easy to block their servers en masse and forget about them. Some companies spam the DNS namespace with predictable, but extremely numerous domain names, which are easy to block using a few regular expressions. Try to make filters in gmail for that, if you don't know from which of the 100 domains the next email will come.

Email from hacked servers is also easy to block. It's mostly PHP servers and all you need to look for is mention of eval() in the headers as nobody sane hopefully evals PHP code to send email.

It just took me a month of spending a few minutes every other day analyzing headers of odd email or two which passes through some generic checks like checking if sending IP address has a domain name and figuring out how to block the sender entirely if possible.

Now I don't get any legitimately looking spam at all and what I get is easily filtered with bayes filter in thunderbird.

Anyway, with spam the hard job is checking the spam folder and that's annoying as hell with gmail, because it's always full of crap, and it's not easy to see occasional false positive. Now I only get 1 spam every two to three days and that's easy to check. Legitimate people who get blocked get bounce message immediately and have chance to re-send according to instructions in the bounce, instead of falling into spam folder and feeling ignored.

Much better experience overall.

[megous]

Actually what is hardest to filter is bounces from gmail servers. I'm not really sure how spammers generate them. They are not in response to anything that I send. It seems like google ignores my SPF records, even though it indicates that it found that the sender forged the From header and sends me the bounce with attached spam that is targeted at me anyway. Quite annoying.

EDIT: I guess I can just reject the gmail bounce if it contains the "Received-SPF: fail (google.com:". Ah!

[cuspycode]

I agree. I have been self-hosting my personal email since 1998, and there was a period when this was difficult due to technical issues related to encryption. But for the past decade or so those issues are gone. The benefits are great. For example, being able to block entire netblocks at the routing or firewall level is an amazing anti-spam tool that is completely free when you self-host.

[mahyarm]

How do you deal with 'important mass notices' from your utility company or bank? Do they only use their own email servers?

[SomeCallMeTim]

> Google doesn't lose my email; I probably will lose my email, because an email server is backed by a database and doing database backups right is hard if that is not your day-job.

I use IMAP email. My email is simultaneously stored on my server and on every client. If the server is nuked, I can set up a new IMAP server elsewhere and sync my email client to it; I'd want to do this from work where I have gigabit internet, or this would take a while, but it can re-upload all the data to the server.

That said, I'm using a managed account. I'm not communicating about anything that I care if the government subpoenas, and I have no plans to.

Unless we end up in a totalitarian state where constructive criticism of the government becomes an offense. But in that case my public posts would be more than enough to convict me without looking at my emails.

[mrighele]

> ...because an email server is backed by a database and doing database backups right is hard if that is not your day-job.

I store my email on dovecot with Maildir storage. For a single or just a few accounts is perfectly fine and you can backup the emails with your favorite backup tool.

[thomnottom]

I spent 2 years trying to get them to understand that alerts from my credit card company were not spam before finally giving up and moving off Gmail. I am very happy to be done with their spam-filtering.

[chmars]

I use filters in such cases – they have worked fine for me so far.

[djsumdog]

> Gmail's search and spam-filtering are both very good

Google/Microsoft's spam filtering makes it impossible to send e-mail from a self-hosted solution.

http://penguindreams.org/blog/how-google-and-microsoft-made-...

Unless you're sending out thousands of e-mails per day and build your reputation with their magic-goo trust filter algorithm, you cannot run your own e-mail server and run with the big players. They have made self-hosted e-mail totally unreliable.

I think what you meant by "very good" is "piss fucking terrible."

[fixermark]

Not for my use-case. There's basically nobody self-hosting email that I want to receive emails from. It turns out the egalitarian "Everyone is an Internet admin" solution favored the spammers heavily over the technocrats or common users; letting Google build a system that defaults to trust-off for self-hosting proved to be valuable for a lot of people.

(Because if a tech-savvy user really wants to email me, they know how to make a throwaway email account and sign the correspondence with a verifiable PGP key).

[snuxoll]

I haven't had any particular issues getting past spam filters, it certainly takes some time to build IP reputation but in general with nothing more than SPF and RDNS properly configured my mails get through. I really should get DKIM/DMARC working eventually, but my current email solution (GroupWise) doesn't support it natively so I'll have to do some nonsense for that..

[codezero]

I had this problem self hosting but was able to remediate it by making sure my server was doing all the smart modern things like dmarc etc... there are some good resources on HN from others who've set up all the right things.

Of course, this all happened after I got bitten during a job search and had most of my applications hit spam folders ಠ_ಠ

[djsumdog]

If you read the post I linked, I have the correct DMARC, SPF and DKIM records and signatures happening. If I send them to my old University (google) account, I see all that get verified and correct. It doesn't really help.

I suspect part of it might be that it's on a Linode and might be sharing a subnet with other spammy machines. That's probably why MailChimp owns a class C and refuses to sell any of it.

[codezero]

Interesting. I am on Linode as well, that sucks.

Do you host an https site on the same domain? Is your mail server responding to ipv6? (I hear this can be a problem)

[ryandrake]

Can you recommend a good resource for "how to set up your mail server like it's 2017" for those of us who would like to self-host but don't want to spend 6 months figuring it all out?

[codezero]

From recent memory this post covers all the stuff I did with my server: https://news.ycombinator.com/item?id=11946756

There are a lot of testing tools you can run mail through as well to see how well you score.

[24gttghh]

You may find this interesting:

https://mailinabox.email/

[stuckagain]

If the feds want your "self hosted" emails they will break down your door, shoot your dog, and take every computer-resembling object on the premises.

[confounded]

I'm amazed that

"Nothing to hide, nothing to fear"

Has become

"Autonomy is pointless, resistance is futile"

In about a month.

[benevol]

It hasn't.

You need to remember the fact that already Snowden's revelations have proven that the NSA and other government agencies all have specific budgets for astro turfing activities (manipulating the public opinion by massively participating in online discussions).

And a couple of days ago, there was a nice post on Reddit's front page summing up the situation on Reddit. Reddit is basically completely compromised by whoever has lots of money (government, big industries, etc). Any company can buy astro turfing services nowadays.

So no, you can't trust public online discussion anymore. Not on Reddit and not here. Unless for topics you are absolutely certain that no economic interest is part of the equation.

[ceejayoz]

Ah, but what if this post is anti-Reddit astroturfing?

[aw3c2]

Exactly! They have to have good reason to do so. With "the cloud", they can just use their dragnet storage.

[sbov]

Yes, rights aren't absolute. If the governments wants your data on a self hosted server they need a warrant. In comparison, you have basically zero privacy protections when your data is in the hands of a third party.

[stuckagain]

In the case under discussion the government has a warrant.

[derefr]

You could "self-host" on a cloud server in, say, China, or Russia, or Iran (if they have any hosting services.)

I mean, the governments of those places will probably snoop your emails, but if their contents have nothing to do with them, they won't care. And they have no treaties with the US to force their hand to turn anything over.

Think of your server as Edward Snowden. What country should it hide in, so the US can't legally get to it?

[usernametbd]

> they have no treaties with the US to force their hand to turn anything over

Sure, but that doesn't mean they won't happily exchange that info as part of a deal with the US, assuming your data is valuable enough.

[sfifs]

You're forgetting the possibility of rubber hose cryptanalysis applied on you. In fact just by hosting in such places, you're probably inviting more attention.

[ThrustVectoring]

>the governments of those places will probably snoop your emails

Uhm, how? Gmail supports Transport Layer Security (TLS), and >80% of their emails to and from other providers do as well (https://www.google.com/transparencyreport/saferemail/). Reject non-TSL emails, give the server a public key and tell it to throw away the email plaintext, and the only remaining threat vectors seem like "get rubber hosed into disclosing your private key" and "server gets compromised, causing future emails (but not past ones) to get exfiltrated".

[stuckagain]

SMTP TLS doesn't and can't validate the certs. It is trivial to MITM it.

[st3v3r]

"I mean, the governments of those places will probably snoop your emails, but if their contents have nothing to do with them, they won't care."

Can't you say the exact same thing about the US government?

[confounded]

Iceland.

[wyager]

It's a lot more effort than just asking google. The goal behind practical security is to make things expensive, not impossible.

[donald123]

And will be pissed looking at a bunch of encrypted files.

[megous]

At least you know what happened. With google and such, you wouldn't even know you're under threat.

[throwaway2048]

They can get everybodys gmail with a single request, how many doors are they going to break down, and dogs are they going to shoot?

[stuckagain]

Are we talking about bulk requests? The case we seem to be discussing here involves "data associated with three Google accounts held by an individual who resided in the United States."

[Spooky23]

Yup. But you'll have more due process.

[wyager]

I recently switched from gmail to ProtonMail. Not self-hosted, but (theoretically) encrypted while at rest using a GPG key derived from my password. Definitely an improvement. If you really want SMTP, as I understand it they have an "SMTP bridge" software that you host yourself that uses ProtonMail as a backend. Seems like a good compromise. You don't have to worry about constant uptime or disk failure, but your data is still fairly well protected.

[megous]

You probably don't need constant uptime. Sending server will try to deliver for a day or two, usually. Legitimate servers use sending queues.

[uladzislau]

Especially when Gmail is not as reliable as we used to think. Gmail DOES loose emails.

http://uladzislau.com/gmail-emails-missing/

[stuckagain]

Lunatic conspiracy theories about what Google does with email abound. There are several likely and reasonable explanations for why you can't find the mails you expect. Two of the most likely being 1) the messages were never acknowledged with a 220 response from gmail's smtp protocol translators to begin with; 2) the messages were accepted but are still in flight for some reason. Gmail does not accept and then silently drop messages. If they are accepted, they will be delivered.

[pklausler]

> Gmail DOES loose emails.

Sounds like a tagline for an erotic movie, that does.

I think you meant "lose".

[frogpelt]

To be fair, English is difficult.

Lose could rhyme with rose or close. But it doesn't. Instead it rhymes with cruise, clues, two's, moos, and choose (almost).

[Navarr]

I think Google fought very hard for it's case.

"Self Hosted" is certainly not going to fare much easier in this same situation.

I do not think that the same result would be had if Google was refusing to deliver data on a non-US server for a non-US citizen.

[solotronics]

I think that in this specific case Google was able to push back much harder than a typical individual would. Who can afford a team of world class lawyers to go up against the fed?

[SEJeff]

And they can still appeal potentially up to SCOTUS.

[Spivak]

Self-hosted would absolutely fare better in this situation. It's not perfect, but at least you would know you're being investigated. This whole mess is predicated on the fact the government is allowed to request your data from Google without much fanfare because they are technically in possession of it. The DOJ was able to successfully argue that user emails are actually business documents owned by the email provider.

This breaks down when the person they are investigating is also the email provider.

[sfifs]

> It's not perfect, but at least you would know you're being investigated.

And this basically is the reason multi national companies self host email servers

[Navarr]

You'd have to own the server too, at that point.

If you were a VPS or even a dedicated lease, or shared - I think the fed would be able to pull the same thing.

[megous]

Not necessarily. You can use POP3. If you pickup often they would find precisely nothing.

[donald123]

Doesn't matter if you encrypt your emails, since you have control of your server. Let the fed have the encrypted data.

[fixermark]

... at which point you might get to rot in jail for contempt until you fork over the decryption key.

I don't have a convenient link to the xkcd comic right now that talks about the difference between theoretical and practical security. ;)

[Roujo]

To quote Spivak higher up in the comment chain:

> It's not perfect, but at least you would know you're being investigated.

[donald123]

Am I gonna be jailed for refusing to decrypt my files?