[mtam]

“We’ve secured the impacted SaltStack service by updating it and adding additional IP filtering, allowing only our servers to connect to it.”

So this means they had Salt master ports publicly accessible? Why would anyone have salt ports open/exposed to public/internet?

[dijit]

> Why would anyone have salt ports open/exposed to public/internet?

If you're bootstrapping random servers, this is a fine approach.

The whole Salt connection methodology is 'trust on first connect' (a bit like the default SSH) with a manual stage in accepting an incoming request and the connection stream is encrypted.

If you're using salt to bootstrap your VPN servers or network appliances then it's understandable that you'd have it exposed to a more public network, and the documentation was clear that this was fine.

Not everything is a virtual machine on a cloud provider.

[alexandercrohde]

Kind of a tough situation. I personally wouldn't be ready to accept this is the last such vulnerability that will be found.

In light of this attack, maybe going forward have a setup script that creates an SSH tunnel back to a machine that can talk to the salt-master for you. You could then have VPN, but if it's flakey at all, it could cost the ability to update machines.

Or perhaps (and I say this as a saltstack user) ansible really is the more secure model for those scenarios.

[darkwater]

> If you're bootstrapping random servers, this is a fine approach.

Define "random". I think there is an alternative method not involving exposing you CM server on the Internet for almost any definition of random. In the Algolia case it's pretty sure because they now filter the access by IP (so they KNOW the IPs)

[dijit]

"Random" can mean "I don't know before I start my instance".

If you're multi-cloud (vultr, DO, AWS and GCP) you almost certainly will not know your instances IP before it's provisioned and you can't make use of nice features like network tags or security labels.

If you're producing test environments then bootstrapping those is going to be significantly more painful than just opening up your salt-master and running an authenticated API request to allow those new machines.

As other people have mentioned, this was always supposed to be /possible/ it's akin to SSH. Sure, you can avoid some log spam and potential issues by firewalling it off- but it's meant to be possible to run it publicly, it has always been marketed this way so it's not "insane" that people did it.

[darkwater]

> As other people have mentioned, this was always supposed to be /possible/ it's akin to SSH. Sure, you can avoid some log spam and potential issues by firewalling it off- but it's meant to be possible to run it publicly, it has always been marketed this way so it's not "insane" that people did it.

I'm not blaming anyone, I'm just saying that if you put well-known software facing the Internet you are exposing yourself to more risks than not putting them on the Internet. And for a core infra software as SaltStack I don't really see a good reason to justify it. I don't justify either putting SSH publicly accessible unless you are a really, really small company or an individual.

[user5994461]

In a multi clouds setup, all the clouds are joined together with site to site VPNs. One doesn't just do a setup where they're public and connect to one another database over the public internet.

[mirimir]

Yeah, that jumped out for me too. I'm guessing that they didn't want to deploy some sort of private network layer.

[lykr0n]

That's easier said then done. There are no simple cross cloud provider solutions for a private networking other then ZeroTier, which has it's own issues.

[kureikain]

Last time I was able to build Azure <-> AWS and GCP <-> AWS use their VPN tunneling and a strongswan server on AWS.

It's only AZure <-> AWS <-> GCP, Azure <-> GCP I didn't try bcuz we just want to connect to central AWS node.

I think IPSec with the "right" config is good enough. But the pain is managing the route tables :(.

[mirimir]

As a hobbyist, I might use tinc or PeerVPN. Or Tor plus OnionCat, with restrictive ip6tables rules. I've used that for a private Docker repository.

But those are probably not secure enough. Or too much hassle to setup.