[ahofmann]

Wow, this article is super smart marketing by tailscale. Not only do they list all the nice and expensive features, that can help in such a situation but they also show that someone at huggingface made a very stupid thing by writing a reusable auth key in an env file. Everyone using mesh VPNs like tailscale, netbird etc. knows that this is like leaving the keys right at the door.

[lukeschlather]

> they also show that someone at huggingface made a very stupid thing by writing a reusable auth key in an env file

I don't see where it says that. The Tailscale key specifically it says was stored in the kubernetes secret manager, and obtained once the attacker already had root on the k8s cluster, so would've had full access to all the secrets stored in a sensible fashion.

They did get root by dumping the environment for a process, but "don't store secrets in environment variables" while it is a valid bit of hardening advice, I wouldn't call it stupid to store a secret in an environment variable.

[itopaloglu83]

> How did you get past security? His fortress is impenetrable. > > Door was unlocked.

Once you’re a root at a system that has the ability to add and remove nodes to a network, it’s pretty much over, at least for being able to add a Tailscale node.

[Rafert]

And in case people are looking for an open source alternative, Fly.io's tokenizer is a credential-injecting proxy: https://fly.io/blog/tokenized-tokens/

[inigyou]

And yet everyone seems to do it anyway. Fine for medium security, but maybe the product needs a high security mode that enforces inconvenient decisions?

[moontear]

Not sure that everyone does this anyway. There are some good security postures you can take with Tailscale as well and leaving an auth key lying around is not one of them.

Tailscale lock should have been enabled. For CI/CD purposes the auth key could have set specific tags, which would result in specific ACLs that limit blast radius. The auth key could have a short validity. You could actually use the Tailscale API to generate alerts when a new device joins your Tailnet and ping your phone or something. You could have a complete separate Tailnet for your CI/CD workers.

And all of that was only the free features.

[AbstractH24]

I see a tailscale product that is basically “middleware for agents” using whatever buzzword you feel (maybe “orchestration”) coming soon

[stingraycharles]

I don’t think everyone does this at all. Maybe very small scale companies, but I would have expected a company like Huggingface to have better security standards.

As soon as you’re big enough to have dedicated devops people, you shouldn’t be doing this anymore.

[inigyou]

You have just been shown an example of "everyone does this". You have gotten moralistic and said they shouldn't do this, but how does this refute the fact that they do?

[stingraycharles]

One example from a root cause analysis of a security incident like this is not evidence for “everyone does this”. I believe that Huggingface, with 250+ employees, is an outlier.

[minraws]

Wait are people really do this so often this isn't even the default tailscale setup method.

I am quite surprised people consider this to be business as usual.

Huggingface for a serious service has never felt truly serious to me for reasons like these.