[LinuxBender]

> Is the attack on its webmail/website or on their smtp servers?

It is an attack on the httpd and smtpd daemons. imap has been unaffected as far as I can tell.

> how does one protect smtp servers from distributed attacks?

By design, MX can be as distributed and in large number as you can afford or as willing to spend. This can be a combination of load balancer virtual IP's distributing load to many MX servers behind it and many MX DNS records with the same or different priorities. This of course won't help much if the people attacking are paying ddos-as-a-service farms to bring on massive volume and packet rates that overload all your servers. There are DDoS scrubbing services you can pay for that will advertise your AS number or use GRE tunnels or VPN's to clean the attack data for you. These scrubbing solutions are no guarantee of mitigation.

> Do tools like fail2ban help?

No. That would be pointless whack-a-mole. If an individual person is mad at you and launching a tool from their own PC or a handful of VM's, then yes fail2ban will help. Blocking individual IP's on your MX servers under a real distributed DDoS attack would be futile. Scrubbing centers are about the only solution once the attack is big enough. Or if you had unlimited funds you could deploy many datacenters or point-of-pressence destinations and build your own scrubbing networks but that is very expensive.

[tgsovlerkhgsel]

I agree that you do need some form of first stage that can take the traffic volume (usually a third party service, but could also be 100+ Gbps of bandwidth you get somewhere and an appliance that can do IP filtering), no good way around that.

But once you have that, you should be able to tell them "forward me all traffic except <list of IPs>" and then list the IPs that are sending you the most (remaining) traffic, or even cost (e.g. if the IP is sending little traffic, but performing many TLS handshakes). That's where a fail2ban like tool would come in, no?

The benefit of this approach is that it works completely independently of the protocol you're running. TCP, UDP, doesn't matter, as long as the attacker cannot spoof IP addresses at scale.

[LinuxBender]

Agreed adding bandwidth can help against some attacks.

as long as the attacker cannot spoof IP addresses at scale

That is kindof the rub. Until a majority of tier-1 backbone providers implement bcp38 [1] or some derivative of it, spoofing from the ddos farms is trivial. There has been talk of implementing this for many years but very little action. Perhaps when DDoS attacks cost enough tax revenue or impact investors, perhaps there may be push for legislation in some countries to implement but in an ideal world most of the providers would work through this as one big team. I just make some network engineers laugh, or smirk, or other

[1] - https://en.wikipedia.org/wiki/Ingress_filtering#Networks

[tgsovlerkhgsel]

My understanding is that the attacker can generate a relatively small amount of IP-spoofed traffic, which is then used to generate a much bigger amount of amplified traffic from servers the attacker doesn't control (i.e. not IP spoofed).

So you need to be able to handle:

a) the reflected volumetric attack (you can block IPs and even protocols/ports here),

b) a much smaller volume of IP-spoofed traffic, which for TCP services basically means you have to be handle a SYN flood (because if they spoof, they can't get a connection established to do the nastier stuff)

c) make sure you don't let your countermeasures against a) be triggered by b) in a way that disrupts your legitimate customers.

For a TCP based service, I'd expect the following to be quite effective:

1. At the very first stage (has to be able to handle the full volume, may be external), discard all UDP traffic, any TCP traffic pointing to the wrong port, and any traffic from blacklisted IPs.

2. At the second stage (has to be able to handle the IP-spoofable volume), terminate TCP connections using SYN cookies, and forward actual connections to the real servers.

3. At the real servers, any remaining incoming connection is coming from a non-spoofed IP. Monitor cost (TCP handshake, TLS handshake, authentication attempt) and successful accounts per IP. Block IPs that cause a high cost but no legitimate users, or an extreme cost and few legitimate users, using the stage 1 blacklist.

4. Optionally you can also aggregate information about successful logins and block sources that cause a high load at stage 2 and have few successful logins using a short-lived entry on the stage 1 blacklist.

UDP based services would be much harder since you don't have such a trivial stage 2, but for TCP, stages 1+2 should be something that you can buy.

[elaus]

Curious as you speak of "very little action", yet the Wikipedia article you linked says:

> […] contrary to general opinion about the lack of BCP 38 deployment, some 80% of the Internet (by various measures) were already applying anti-spoofing packet filtering

I know very little about global routing, so I'm wondering if it's an ineffective way of filtering against these attacks we see today or is 80% not enough to be effective?

[LinuxBender]

It really isn't enough. Everyone needs to do their part [1] not just ISP's but also large businesses, cloud providers, others. And this has to occur in all regions. I admittedly over-simplified the problem statement.

[1] - https://archive.nanog.org/sites/default/files/1_Kaeo_Ddos_Tr... [PDF]

[rgrmrts]

Thanks for your response! So using specific examples here for smtp, I get a 1gbps guaranteed network from Hetzner so in theory I’d need to distribute over 50 servers to withstand this attack?

It’s not clear to me why fail2ban wouldn’t at least help, if the botnet is a thousand machines wouldn’t I (eventually) have them all blocked? And therefore reduce the overall duration of the attack? Or is the problem that it’s hard to differentiate between good clients and bad clients because no single client is sending enough traffic to be suspicious?

Also, do you have any specific examples of ddos scrubbing services? Would like to take a look specifically at affordability for individuals.

[LinuxBender]

In regards to fail2ban, assuming the attack was purely SMTP specific which it won't be even then blocking IP's would be futile. The DDoS-as-a-service farms have hundreds of thousands to millions of IP's under their control. Block one IP and ten more show up. One IP is not one attacker. Those farms have probes that can tell how effective their attack is. Some of them even have "proven work" that is reported back to the buyer to validate the effectiveness of their attack. You may want to research this one as it is a very big topic.

In reality however, a large portion of the attack won't even show up in logs. The attack will also contain depending on how much the attacker is willing to spend tens of millions or hundreds of millions of packets per second of TCP, UDP packets on random ports, no ports, random protocols, random sizes, random TTL's, random headers. That is a volumetric attack. Fail2ban and most network IDS/IDP's would not even see this attack. It would saturate the uplinks to the ISP before you even see anything. Your ISP will most likely null route you and encourage you to stop advertising your AS number.

In reference to scrubbing centers for individuals, that is not a thing unless you have unlimited funds. There are some VPS providers that can scrub tiny attacks. Linode, Vultr, OVH to name a few but they can only deal with tiny attacks. If you want to research this for your business, my suggestion would be to get on the NANOG mailing lists and discuss it with all the network engineers to find out which scrubbing services are currently most effective. This is a moving target and a very big investment. I am not a fan of any of the companies that provide these services, but that is only based on my limited experiences with them.

[rgrmrts]

Word, thanks for all the details! Yeah, I don't have any actual business use-cases outside of a personal interest in making my dedicated server reasonably protected.

[LinuxBender]

You're welcome! For your personal MX servers, the most cost effective solution I know of would be to have multiple domains and each domain have their own MX records and corresponding VM/server on its own unique provider to isolate them. Then ensure that people/businesses that are important to you know to contact you at 2+ email addresses/domains. If someone attacks one of your domains or MX servers, you can safely ignore the attack. These separate MX servers should be on different server/VPS providers in the event that the attack causes one of them to suspend your account. Your imap client can poll each of the servers/domains so that you get your emails.

[rawoke083600]

Would a decenterlized service architecture work better than an server-client arch ?

Im thinking like a lambda/faas platform where the computing endpoint is massively distributed ?

I guess there are (currently) no 'one final solution' but like traditional security a good solution consists of many many layers/rings of defences ? Can one(in theory) decenterlized all 7 of the osi model ?

Trying hard not to use the word (crypto/blockchain) here .

[toast0]

Function as a service isn't really what I'd call decentralized. It's centralized at the service provider. But, they mix your load in with everyone elses and they serve the load in many places (maybe geographically distributed is decentralized?).

But, voip.ms has points of presence all over and they were DDoSed effectively recently; distribution can often help, but it's not enough. If an attacker sends X gigabits of garbage at your San Jose PoP and disables it, your other PoPs will likely function, but if they send X/N gigabits at each of your N PoPs, that might be enough to disable all of them.

Using a large provider can help a lot though. Volumetric DDoS is 'solved' by having large pipes and discarding lots of traffic. And, where possible, getting upsteams to discard lots of traffic before it arrives on your network. Large providers have large pipes and good relations with their carriers. Smaller providers or DIY doesn't.

[LinuxBender]

Email is already designed to be decentralized. You can have a very large number of mail servers in any number of clouds behind multiple MX records, Anycast IP's, Load balancers, etc... The scale of distribution required to defend against these attacks is possible, but prohibitively expensive. Some VPS providers do support Anycast so in theory you could dynamically spin up tens of thousands of inbound MX nodes that would at least spool the emails. Anycast will allow you to have any number of MX servers appear as one IP address. Those nodes in turn then need to relay that to your centralized servers. This brings up more scalability issues as you could essentially DDoS yourself in this pattern. There may not be enough clouds to support this idea and the cost would give just about any CTO sticker shock or medical issues.

If you mean distributing the destination servers that people access that is basically the same pattern. One could in theory spin up cloud instances of the email httpd/smtpd servers but there are still diminishing returns. This also requires the clients to know how to route to the right clusters of servers. Not easy, but doable. Ultimately this is the same problem people run into with web servers hosted on AWS. You can spin up more instances but there are still network link bottlenecks going into each region. AWS and the like have partnerships with scrubbing centers. This model breaks down if the attacker is willing to spend more than $500. Most massive scale attacks cost less than $200 on the dark web. If the extortionists believe they can get more than say $1k, then they might be willing to spend enough to even take down some scrubbing centers. It can cost hundreds of millions to defend against a $200 attack.

The current email RFC's would not support something like crypto/blockchain. At that point you are basically inventing a new standard and adoption may take a very long time unless there is a compelling business advantage to it in my opinion.

[mirashii]

The problem is that fail2ban or any sort of dropping of packets at the host is too late, the volume has overloaded the link.

[deadbunny]

You're thinking too small. DDoSaaS use hundreds of thousands to millions of compromised machines* to attack. You might only see a handful of connections from each IP. Whch makes it fruitless to use something like f2b.

* Machines here could be anything from a lightbulb to a server.