[toomuchtodo]

Who is exposing their memcached instances on the public internet? Who is not filtering outbound UDP traffic from their memcached instances?

Rhetorical of course. Akamai should have logs of their offenders. Off to scan for offenders and notify their providers!

[theojulienne]

During some analysis we did notice that at least some cloud providers default to having instances with public IPs (with no network-level ACLs) by default, and some Linux distributions default to having memcached listening for UDP traffic and binding to `0.0.0.0` by default as soon as it's installed. The unfortunate combination of these result in the machine being vulnerable to being used as an amplification vector in these attacks.

[snuxoll]

This is one area where I really disagree with Debian and derivatives default behavior of starting a service immediately after installation and not having a firewall enabled by default.

If I install a service on CentOS/RHEL/Fedora it is disabled by default, if I start the service firewalld will block traffic until I have explicitly enabled a rule to allow it (or explicitly stopped and disabled the firewalld service).

Does this prevent people from making poor decisions, like just blindly starting the service without reading the configuration file, or disabling firewalld/enabling a rule without checking the configuration first? No, it doesn't - but that small hurdle at least prevents people from inadvertently turning on a service and opening it up to the world just by installing a package.

[vbernat]

As a compromise, Debian also comes with a secure-by-default configuration. In the case of memcached, the service is configured to listen on 127.0.0.1.

[snuxoll]

Thankfully that is one thing they do pretty well in most cases, though there are some (apache2, for example) that do listen on all interfaces by default. While even services like Apache may have secure configurations by default, it can often be installed by other programs that link or copy their own configuration files into the apache config directory - and then all you need is a PHP vulnerability or whatever.

[toomuchtodo]

Did you receive any cooperation from those cloud providers in using ACLs at their network edge to drop that traffic (that they should’ve been blocking in the first place)?

[toast0]

> Who is not filtering outbound UDP traffic from their memcached instances?

This is of course the wrong way to do it -- you need to filter inbound UDP to your memcached instances so you don't waste your resources generating the responses, and also so you don't accidentally fragment the responses and only drop the first fragment outbound.

[toomuchtodo]

I disagree, due to seperation of responsibilities. Having run both an ISP and a hosting company, you have to filter traffic at your edge that can impact external resources (just as ISPs block outbound NetBios and SMTP traffic on port 25/tcp).

Yes, the server or instance customer should be doing this. But they’re not, because poor security practices are an externality, not a cost they sustain.

Security is more important than developer velocity, but users pay the bills.

[hueving]

Your confused if you think it's the clouds that are misconfigured here. The issue is the ISPs allowing the spoofed traffic going towards the memcached servers.

[toomuchtodo]

If you’re a service provider allowing your equipment to participate in an amplification attack, you’re the fool trashing the commons.

It’s an ISPs job to filter outbound udp on arbitrary ports? Shall we only let 443 tcp outbound from eyeball networks?

[hueving]

Any UDP service can be used in an amplification attack. It's not the responsibility of AWS or other hosting companies to shut down all UDP traffic.

The problem is the ISPs allowing spoofed IPs.

[mnw21cam]

So the problem here is that a number of UDP packets were sent from somewhere (with a small bandwidth) that had a spoofed source address. They were then sent to the reflection servers which produced more/bigger UDP packets that did not have a spoofed source address.

So the attacker only needs to find somewhere on the internet that is capable of generating spoofed packets. They needed a lot of places that had a reflection server, but the requirements for the spoofing was much smaller.

In other words, you would have to prevent 99.9% of the internet from being able to spoof source addresses before you fixed this problem.

[dylz]

udp is also used for web traffic, a very very large amount of web traffic - so you can't really block it.