[ocdnix]

Fascinatingly effective, but maybe I'm the only one getting the heebie-jeebies when someone suggests implementing this in production corp networks. Sure it's super convenient, but the thought of bypassing all traditional NATs and firewalls, and instead relying solely on a software ACL, seems super risky. Maybe I just don't understand how it works, but it seems that a bad actor getting access to a stray VM with Tailscale on it in, say, your AWS testing env, essentially has an clear path all the way into your laptop on the internal corp network, through the kernel, into user space and into the Tailscale ACL code as the sole arbiter of granting or blocking access. Would I even know someone unauthorized made it that far?

[dfawcus]

That is why many of us keep repeating that NAT is not a security mechanism.

Punching through NAT, and most associated state tracking filters, is very easy.

I've implemented such in a production corp environment, as a product to be sold. There is no magic here, it is all well understood technology by the practitioners.

If you actually want to have packet filtering (a firewall) then deploy a firewall instance distinct from any NAT, and with appropriate rules. However that only really helps for traffic volume reduction, the actual security gain from a f/w per se is now minimal, as most attacks are over the top: HTTP/HTTPS, POP/IMAP etc.

[cyberax]

> That is why many of us keep repeating that NAT is not a security mechanism.

You can say that in general, network firewalls are not a security mechanism. They are at most a means to prevent brute-force attacks from outside of the network.

[dijit]

to be completely fair with you, everyone misinterprets NAT as a security mechanism, because traditionally it is deployed alongside a stateful firewall.

In reality, of course, the stateful firewall is doing all of the heavy lifting that NAT is getting the credit for. Tailscale does not get rid of the firewall in fact it has a much more comprehensive setup based on proper ACLs.

Though I’m definitely the first to admit that their tooling around ACL’s could be significantly improved

[tptacek]

I think they mostly interpret NAT as a security mechanism because that's what it originally was; "NAT" was a species of firewall, alongside "stateful" and "application layer". And NAT obviously does serve a security purpose; just not the inside->out access control function we're talking about here.

[avianlyric]

> think they mostly interpret NAT as a security mechanism because that's what it originally was; "NAT" was a species of firewall

That’s simply wrong. NAT is, and always has been for the sole purpose of Network Address Translation, I.e. allowing a large IP address space to hide behind a much smaller IP address space (usually a single IP address), for the purpose of mitigating IP address exhaustion.

NATs were meant to be a stop gap solution between IPv4 running out, and the rollout of IPv6. But we all know how that panned out.

The “firewall” like aspects of a NAT are purely incidental. The only reason why a NAT “blocks” unsolicited inbound traffic is because it literally has no idea where to send that traffic, and /dev/null is the only sensible place to direct what’s effectively noise from the NATs perspective.

The fact that NATs shares many of basic building blocks as a very simple stateful firewall is just a consequence of both NATs and firewalls being nothing more than stateful packet routing devices. The same way any standard network switch is (they internally keep a mapping of IP to MAC address of connected devices based of ARP packets, which incidentally blocks certain types of address spoofing, but nobody calls a network switch a firewall).

[tptacek]

You're trying to piece this together axiomatically, but you can just read the history of the Cisco PIX firewall to see that the story is not as simple as you want it to be. One of the first and clearly the most popular NAT middlebox products of the 1990s was a firewall, and Cisco made a whole big deal about how powerful NAT was as a security feature.

[avianlyric]

You’re working backwards here from Cisco’s marketing material. Just because someone in Cisco’s marketing team was smart enough to realise they could market NAT as a security feature, doesn’t mean it was designed to be a firewall.

Apple advertises their iPads as “computer replacements”, that doesn’t mean the iPad was originally designed to be a computer replacement, and it certainly doesn’t make iPads a good computer replacement for many people.

I would also highlight that Cisco PIX had a dedicated firewall layer in addition to its NAT layer, which provided much more capabilities than the NAT layer alone. The fact that these two layers intelligently built on each other is just good implementation engineering, it doesn’t change the fundamental fact that NAT isn’t, and never has been, a proper security tool.

[tptacek]

I'm working forwards from my experience at the time as a security engineer working with products that claimed NAT was a security feature, since it allowed for machines to access the Internet without being routable from the Internet for initiated connections, which is why the first commercial PIX product, after Cisco bought Network Translation (which named PIX), was a firewall.

People confuse the fact that NAT is not an especially powerful or coherent security feature with the idea that it isn't a security feature, which leads you to the crazy rhetorical position of having to argue that PIX, the first mainstream NAT product, was not a security product. I have friends who worked on PIX, for many years. I assure you: they were in the Security BU.

I think this position is pretty hopeless, though if you want to drag us around through the network security marketing of the mid-1990s, I'm happy to do so, just for nostalgia's sake. NAT is absolutely a security feature, and was originally deployed as one, in an era where it was still feasible to get routable allocations for individual workstations.

[cesarb]

> The same way any standard network switch is (they internally keep a mapping of IP to MAC address of connected devices based of ARP packets, which incidentally blocks certain types of address spoofing, but nobody calls a network switch a firewall).

I thought standard network switches kept a mapping of MAC address to physical network ports, and didn't concern themselves with the IP layer at all (other than things like IGMP/MLD snooping)? Mapping from IP to MAC addresses is a function of hosts/gateways, not switches.

[yencabulator]

Lots of switches filter out ARP responses that would change the destination of traffic to preexisting clients.

For example: https://www.arubanetworks.com/techdocs/AOS-S/16.10/ASG/YAYB/...

[dijit]

I mean, it really isn’t a security mechanism of any kind. Any security properties at all are completely accidental.

One need only disable stateful firewalling and use that to see how completely dire the situation would be. As all outbound connections open up your host to the internet.

[Hilift]

> production corp networks.

Networking has long been the toxic wasteland of security and misconfiguration. Now combine that with newer host-based networking models for containers. The Windows network stack is substantially different now due to that, and more complex. Since Wireguard has been part of Linux, everyone and their brother now has a VPN, somewhere connecting to a VPS. It's probably worse than you think because you don't know what you don't know.

[rixed]

This is to go through NAT, which are devices designed to work around the rarefaction of IPv4 addresses.

Firewalling is a different concept, but since you raise that issue of connectivity wrt. security, I have to say that what makes /me/ sad and anxious is to see how internet security has always been hinging on bloquing paquets based on the destination port.

Doing what's easy rather than what's correct, exemplified and labelled "professional solutions"...

[navigate8310]

I'm rather more curious as to why you stylized "bloquing paquets"?

[aborsy]

Maybe the OP is French? :)

[rixed]

Haha indeed.

I also frequently stumble upon "connexion" and "trafic", in the same department.

[antod]

They are emphasizing the queuing implementation.

[Muromec]

That’s how all voip worked since forever and we also have a bunch of standard and public facing infrastructure to make it easier. All the ice, turn and friends.

It still needs something on the inside to talk to outside first, so the actual firewall should whitelist both outbound and inbound connections.

Than again, if you rely on perimeter, it’s a matter of time when someone figures out what’s your equivalent of high wiz jacket is.

[totallywrong]

It's no different from traditional VPNs. The tailnet admin has control over the routes that are exposed to clients and ACLs are available to further limit access. It's an overlay network, it doesn't magically give you access to user space on people's laptops.

[ray_v]

Given how tailscale works and many of the features (the SSH features especially) it's not terribly hard to imagine a critical flaw or misconfigured setup giving access to userspace

[avianlyric]

Everything beyond tailscales core VPN features are opt-in. The risk of misconfiguring Tailscale is the same (arguably it’s much smaller) as just misconfiguring SSH on a machine.

At the end of the day, Tailscale works just like any other VPN, from the perspective of the type of data that can traverse between machines connected to the same virtual network. Tailscales use of a P2P wireguard mesh is just an implementation detail, it’s no more or less risky that having every machine connect to a central VPN gateway, and bouncing all their traffic off that. Either way, all the machines get access to a virtual network shared by every other machine, and misconfigured ACLs could result in stuff getting exposed between those machines, which shouldn’t be exposed.

If anything the Tailscale mesh model is much more secure, because it forces every device to use a true zero trust model. Rather than the outdated, “oh it managed to connect to the VPN, it must be safe then” model that traditional VPNs often end up implementing.

[rainsford]

I'm not sure how to compare the risk and attack surface of traditional NATs and firewalls vs Tailscale's ACL code, but I'm not sure Tailscale is obviously the riskier choice there. I think more traditional network devices are more familiar and more of a known quantity, but there's a lot of janky, unpatched, legacy network devices out there without any of the security protections of modern operating systems and code.

It's also worth considering that exploitability of ACL code is just one factor in comparing the risk and Tailscale or similar solutions allow security conscious setups that are not possible (or at least much more difficult) otherwise. For example, the NAT and firewall traversal means you don't have to open any ports anywhere to offer a service within your Tailscale network. Done correctly, this means very little attack surface for a bad actor to gain access to that stray VM in the first place. You can also implement fairly complex ACL behavior that's effectively done on each endpoint without having to trust your network infrastructure at all, behavior that stays the same even if your laptop or other devices roam from network to network.

Not to say I believe Tailsclae is bulletproof or anything, but it does offer some interesting tradeoffs and it's not immediately obvious to me the risk is worse than legacy networks (arguably better), and you gain a lot of interesting features and convenience.

[miki123211]

And for whatever it's worth, Tailscale is written in a language that makes buffer overflow and memory corruption vulnerabilities extremely unlikely.

[bradleyjg]

You don’t want to be hard on the outside, soft on the inside. Especially because you probably aren’t that hard on the outside!

Defense in depth.

[devmor]

That is a whole lot of different levels of exploits that would have to be chained together that you just trivialized there.

How do you suppose they gained access to the kernel and userspace just by having a network connection to the laptop?

[Muromec]

By using an unpatched rce in any network exposed code. The whole point of firewall is to prevent bad hackers from the bad internet to exploit your unpatched rces, abuse your default passwords, host based security you shouldn't have had in the first place and access stuff using compromised credentials you didn't revoke or didn't know you should have revoked. Because consistently doing all of that all of the time is hard for creative professionals. It's a chore. It's a tax.

[avianlyric]

How exactly is Tailscale different to literally any other piece of network capable software in that regard?

NAT traversal requires careful coordination between two devices to create a connection, it’s not like any random device on the internet can perform NAT traversal against a machine just because it’s running Tailscale (not to mention every modern browser has NAT traversal built in for WebRTC connections).

And if the issue doesn’t arise from using NAT traversal, then how does Tailscale expose anything more significant than what a traditional VPN will expose? After all the only difference between a P2P VPN and a traditional VPN, is that a traditional VPN bounces all your traffic off a common server, rather than attempting P2P connections.

[cyberpunk]

I think the point is not that there are necessarily exploits, but by compromising one node in the tailnet they now have the ability to hit code in these locations, or services running on your tun0 interface on your laptop etc.

[avianlyric]

How is compromising a a single node in a tailnet more dangerous than compromising a single node in a traditional VPN?

Traditional VPN don’t usually put firewalls between machines on the network, because traditionally the whole point of a VPN is to avoid the need for firewalls to provide security between nodes on the virtual network, by assuming that only safe machines can connect to the VPN.

[aborsy]

You would typically remove the default any to any ACL rule, and allow the connections that you need. The compromised node normally would not have access to anything interesting. Normally it’s jailed, or would not be able to make outgoing connections.

Am I missing something?

[cyberpunk]

The ACL logic happens in the tailscaled on the destination though doesn't it? So even if you block the access via the ACL the packet has still gone through the network stack and go runtime etc before the traffic is dropped which is a significantly bigger surface than a (traditional) external network firewall.

[aborsy]

I see your point. You are talking about a vulnerability.

You are right. Tailscale nodes can send packets processed in any other node, irrespectively of ACLs. Essentially each node gets to “run code” in other nodes, which is normally dropped. I don’t know how deep the Tailscale packets go before being dropped (perhaps the coordination server distributes firewall rules).

But you have to compare with another access method, like, the hub and spoke VPN. The compromised and uncompromised nodes connect to a VPN access server. A compromised node sends packets that are processed in the VPN server, but can also connect to the uncompromised node, meaning, the latter has to process and drop the packets of the former. You have to trust the OS IP stack. To some extent, the same is true if the trusted node VPNs directly to the untrusted node. During an established connection, the networking stack of the trusted node has to block the other side.

Maybe someone familiar with the implementation of ACLs in Tailscale could chime in.

Update: The ACL rules are applied to the incoming packets over tailscale interface. The filtering is then done by tailscaled. The packet has gone past the interface and processes by tailscaled. So an unauthenticated packet indeed travels through the kernel space all the way to the userspace.

[avianlyric]

How is this any different to any other piece of network capable software that’s listening to a port on your machine?

An external network firewall can only offer protection if you can somehow guarantee that every packet that hits a specific node is first routed via that firewall. Traditionally nobody has setup networks like that, because it requires routing every single packet via a single common bottleneck, causing huge latency and throughput problems.

As for packets going via the network stack, and then the go runtime. Do you honestly believe there’s set of vulnerabilities out there which would allow random external packets to be sent to a random machine, and cause an RCE by virtue of simply being process by the OS kernel, which somehow can only be exploited if you’re running Tailscale? Better still, if such a vulnerability exists, what on earth makes you think your firewall isn’t also vulnerable to same issue, given that pretty much every firewall out there is built on the Linux kernel these days.

[cyberpunk]

It goes into tailscaled before it's dropped, and tailscaled has had a number of security issues in the past (including RCE):

https://tailscale.com/security-bulletins#ts-2024-005 https://tailscale.com/security-bulletins#ts-2022-004

[kortilla]

> Am I missing something?

Yes

> You would typically remove the default any to any ACL rule

This part doesn’t happen.

Defaults are rarely changed.

[0xbadcafebee]

Network security is a myth. NATS, firewalls, ACLs, etc don't keep you safe. Even on your Wifi LAN right now, you aren't safe from local network attacks originating from outside attackers.

[dd_xplore]

Why?

[0xbadcafebee]

Because hackers can contort themselves into amazing shapes in order to fit through tiny holes in the oddest places. Once they position themselves correctly, and are able to reach the network address and port of a given service, and it has no authentication, it's open season. It may seem difficult, nigh impossible, for a hacker to reach all the way into your WiFi LAN. But there are always twists and turns to take.

From the public internet: tens of thousands of internet routers have publicly known exploits right now, which the router vendors refuse to fix. Just scan the internet for the routers, use your exploit, and you're inside.

From the opposite direction: malware in a website can redirect your browser to the management interface of a router on your local LAN, where it can reconfigure your router. If there is a password but you have logged in from your browser, the active session token lets it right in, and CSRF protection is often disabled or incorrectly set up. And even if it has a password, many such routers have exploits that will work despite a password. Many people also fall for phishing attacks that can drop payloads on your machine directly.

In some cases, the ISP itself has shipped a firmware update to routers that included malware.

All of these things have happened in the past 2 years, to millions of internet users, that we know of. Many large attacks go unnoticed for years. Once the router is compromised, it can be configured to forward ports or enable UPnP, or simply persist malware inside the router itself. The network is wide open and at the attacker's fingertips.

And this is just one class of attack. There are many more that can attack private networks. So there is no place safe from network attacks. Not in a corporate network, not on your local LAN, nowhere. There is no network security. The only network services that can be somewhat trusted are ones which require strong authentication, authorization, and encryption.

[avianlyric]

A better question is, “why do you think your local network is safe?”.

Have you taken steps to validate the integrity of every single device connected to the network?

If a single device is compromised, how will detect its been compromised?

If a device is compromised, what prevents it from being used to launch an attack on other devices in your network, especially if your security model assumes that all devices on your local network are “safe”?

For a practical example of this happening, in a very impressive manner: https://arstechnica.com/security/2024/11/spies-hack-wi-fi-ne...

For a more boring everyday equivalent, just search around for one of the many botnets that are assembled from compromised SoHo routers, or IoT devices, around the world.

https://arstechnica.com/security/2024/01/chinese-malware-rem...

Assuming a local network is safe and secure is foolish. There’s nothing inherently secure about a local network, the only reason it offers any level of security is due to a local network being many-many orders of magnitude smaller than the entire internet. So the probability of a hostile device (whether intentional installed as hostile, or became hostile after a remote attack) being connected is smaller. But at the end of the day, is security via “being luckier than the next dude”.

[miki123211]

As far as I understand, Tailscale won't even let you initiate a connection (or give you WireGuard keys for a node) unless there's an ACL rule that allows it.

[kennes913]

Currently evaluating tailscale as a VPN-like solution and read the same thing:

"At a less granular level, the coordination server (key drop box) protects nodes by giving each node the public keys of only the nodes that are supposed to connect to it. Other Internet computers are unable to even request a connection, because without the right public key in the list, their encrypted packets cannot be decoded. It’s like the unauthorized machines don’t even exist. This is a very powerful protection model; it prevents virtually any kind of protocol-level attack. As a result, Tailscale is especially good at protecting legacy, non-web based services that are no longer maintained or receiving updates."

Source: https://tailscale.com/blog/how-tailscale-works#bonus-acls-an...

[binary132]

Isn’t this essentially what a VPN does? I mean, that’s what TailScale is built on: Wireguard.

[tantalor]

Security in depth.