[simonw]

> One of those 136 credentials was a reusable Tailscale auth key, used to create new Tailscale CI (continuous integration, used for automated testing) nodes in their tailnet. The agent copied that key into a series of external sandboxes and used it, over several days, to enroll a total of 181 nodes into Hugging Face’s tailnet. Those nodes each received a Tailscale identity tag granting all the access a CI node would get.

This feels like an alerting opportunity. I wonder what the lowest friction way would be for Hugging Face to have alerts if 181 unexpected nodes were added to a tailnet.

[d3m0t3p]

When everyone push on friday, and you have 400 CICD pipeline triggers spawning that many nodes. How do you know if this is unexpected ?

Their cloud compute might be on demande, someone starts training a model and 50 machines are spawned. Knowning when something is unexpected is hard

[acdha]

The one thing which might have helped is the “external sandbox” detail: registering CI instances is noisy but it should be highly consistent so you could potentially alert on use outside of the expected environment (e.g. I have AWS STS sessions bound to a source IP or VPC, which really came in handy when Trivy was compromised).

I wonder whether it’d be possible to do something like export the list of legit K8s instances periodically so you’d be able to look for usage outside of those, but at that point something like that proxy approach would be less work and workload identity federation would be even easier.

[moontear]

True. Just based on the reporting we don't know what's going on, but the article makes it sound like those 180+ nodes were unexpected. Of course you would need to set your own limits to alerting and "what is unexpected" and you may still fail to catch something this way.

[zactato]

Its always CI. Whenever I've worked with good pen testers they always start with Jenkins, know that people stored creds and there and didn't treat as seriously as they treated production.