>According to the publication, multi-tenant environments where non-admin users have permissions to create Ingress objects are most affected by this issue.
Ouch! I guess this means lots of secret/password rotations for k8s admins coming soon. I've always wondered if there are public Kubernetes multi-tenant setups in the wild rather than just ones with admins and developer roles/namespaces.
For whatever it’s worth my startup began life as a multi tenant kubernetes provider. We have a ton of custom controls - and for example do not allow this “custom snippet” directive at all. We spent a very long time locking down as much of the api surface as possible.
I have war stories up the wazoo - at one point we had 16k random internet users on a single cluster!
(Also, yeah, it'd be nice if the URL of the OP had just been the Github issue. It is just as informative (or uninformative), and isn't an advertisement…)
Although this particular vulnerability is all about ingress-nginx and has nothing to do with Kubernetes itself, the current design of the ingress API sort of encouraged it.
The reason that an ingress controller need permission to read all secrets across all namespaces in the first place, is because ingress objects can specify their own certs (stored in namespaced secret objects) within their namespaces, hence the ingress controller needs permission to read those secrets and load the certs into the reverse proxy (in case of ingress-nginx, it is nginx). This is such a scary permission to grant that cluster admins have to hand over all the secrets in the cluster to the ingress controller and hope it does the right thing.
Now, ingress-nginx actually allows you to specify a default secret to be used by all ingresses it manages, and that would be the only secret it needs access to. I am not sure how many users out there are willing to dig into thousands lines of its manifest yaml files to tweak its rbac settings.
Agreed, I'd prefer a GitHub issue over an advertisement. The HN user who posted this works for the company that wrote this post, and that person's title is "VP Marketing & Business Development".
The issue contains a comment where they're also advertising their service, so I think that using the GitHub issue would be better, and would be a good compromise for them since they'll still have their ad.
The problem is in intersection of nginx-ingress and kubernetes. Since ingress controller has access to secrets from all the namespaces (which is kubernetes side of the story) the nginx implementation with snippets added by users (the nginx contribution) may expose these secrets.
The post also points to an open source tool that helps people to check if they are vulnerable, whether they want to get to the bottom of the issue or not.
I know nearly nothing about ARMO, and I have nothing against you, the company you founded and are CTO of (ARMO), or ARMO providing an open source tool. I appreciate that you have made this work open source.
Still, some of us on hackernews dislike advertising. I 100% agree with everything you've written in this comment, but as I mentioned in the comment you're replying to, the GitHub issue also contains an ad for the same tool - it just doesn't make it front and center, allows for any competitors to make their own comments on an open platform, and it feels less biased. Less advertisey.
Instead of "install our product", it could have been something like "run this thing specific to this exact vulnerability that does the absolute bare minimum. Oh, btw, if you liked that, you should check out our product that does much more, kubescape"
"... a user who can create or update ingress objects, can use the custom snippets feature to obtain all secrets in the cluster." I'm not a Kubernetes guy, so I'm curious, how often is there a user with only those permissions? Is it common to have a user that can create/update and then doesn't already have some kind of other access to everything else? I don't know much about Kubernetes permissions.
(I also just learned that the word Kubernetes is in the default Chrome spell checker thing, which was sure helpful in writing this comment!)
A user can have many permissions in his/her own namespace without compromising the other tenants. This sounds like it would allow any user with Ingress create/update access in any namespace to compromise secrets across all namespaces.
It is unclear whether this is a problem if you're running namespaced ingress-controllers though. This comment [0] in the bug report says:
there's definitely an attack path that gets the ingress-nginx service account token, which has list rights on secrets at a cluster level (so allowing for all secret values to be retrieved).
I can't see how list permissions would allow retrieval of the secret value though. You'd need get permissions for that.
> "... a user who can create or update ingress objects, can use the custom snippets feature to obtain all secrets in the cluster."
To this point, handling of secrets in the Kubernetes ecosystem always seemed like a mess to me. Both `vault` and AWS SecretsManager/KMS in combination with IAM allow me to finely partition access to secrets by instance or developer identity whereas in the Kubernetes world one expects "secrets" to be handled by a small core group and machines seem to have mostly unfettered access.
Specifically[1],
>> Caution:
>> Kubernetes Secrets are, by default, stored unencrypted in the API server's underlying data store (etcd). Anyone with API access can retrieve or modify a Secret, and so can anyone with access to etcd. Additionally, anyone who is authorized to create a Pod in a namespace can use that access to read any Secret in that namespace; this includes indirect access such as the ability to create a Deployment.
>> In order to safely use Secrets, take at least the following steps:
>> * Enable Encryption at Rest for Secrets[2].
>> * Enable or configure RBAC rules that restrict reading data in Secrets (including via indirect means).[3]
>> * Where appropriate, also use mechanisms such as RBAC to limit which principals are allowed to create new Secrets or replace existing ones.
When you dig deeper, both [2] and [3] look like they are someone's job security plan. No wonder it was developed at a place which employs people just to keep them off the market.
Especially since you really don't gain much unless you also have a KMS Provider[4].
No wonder most K8 setups I've seen still have BASE64 encoded secrets in some YAML file.
There is also no shortage of tutorial-spam -- so unless you happen to be lucky enough to apprentice with people who know what they are doing, it is hard to gain a solid understanding and move confidently.
I think it's contrary to currently-recommended ops practice (clearly imperative rather than nominally declarative), but how is it bad security practice? If you don't trust the origin, it's not like saving to a temporary file first is going to help you. Unless you're suggesting that everyone should always do their own code review and then compile from source...
Companies / organizations / team members go rogue sometimes. We've seen this even recently with e.g. kicad, freenode, the timezone database, etc.
Just because you trust the source doesn't mean you should trust all of the scripts they tell you to run. Even if it's a good-faith script, you have no idea if it's making assumptions about your system that are not true and opening you up to sidechannel attacks and the like.
Curl to a file first, inspect the script, consider it within the context of your own system, then run it if you deem it's safe.
Here's some discussion about that. But to put it simply, security is done in layers. Checking something simple like a script before running it is relatively easy and can catch low-effort malicious attempts. Sure, it won't protect against an advanced persistent threat doing a supply chain attack, but you're probably much more likely to be hit with low effort attacks that could be avoided by just not being careless.
At minimum you could download software that is at least signed by someone you trust. Rather than streaming arbitrary commands from a source that the extent of the authentication is that their CDN's TLS certs matches their domain.
Saving to a file first means you can at least run it in a test environment first, then be confident you’re running the same thing when moving into production.
You can check for this vulnerability using the existing tooling any k8s admin already has on their systems by necessity. It's always foolish to install unknown software and security professionals should never advise that.
Looking at the blog URL and header bar, and the script URL, this is pretty clearly a company blog recommending to use the company's own product. I hardly think that context counts as "unknown".
Lets say (for example) that was published on a wordpress site, and the admins for whatever reason didn't secure it properly.
The article in question which _today_ looks all legit and points to a nice working script, might tomorrow be pointing to someone else's script that's a lot less legit.
And yes, in this imaginary scenario that wordpress install leads to a host of other problems for the company.
I will argue until the day I die: curl | bash is actually more secure than most traditional software delivery mechanisms. Slightly, but meaningfully.
With curl | bash, the URL of the bash script is right there. I can copy it, plug it into my browser, and inspect with ease the shell script that will run. If its on Github or a similar site, I can see in plain language the exact organization who published it (I wouldn't implicitly trust whoever "armosec" is, but I would trust, for example, "kubernetes", "microsoft", or "google" e.g.).
Alternatives?
- For something like this, to generate a report on whether you're affected? Maybe just a copy-paste bit of code? Isn't that the same thing, but less convenient?
- apt-get install something. Who published it? Does apt have security scanning (no lol)? Where's the source code for that package (scattered to the wind no doubt)? Does it have reproducible builds so I can be certain the code I eventually find is what I install (lol no. but bash scripts do!)
- Let's talk about `snap` for a second; Did you know that there's a `snap install aws-cli` package available [1], published by "Amazon Web Services", with a verified checkmark. Sounds great. Except, for years, no one on the actual aws-cli team had any idea where it came from [2]. They didn't authorize it, and they don't maintain it. It turns out, it was probably the result of a hackathon project by one AWS engineer. "Better security practices" for sure!
The only legitimate risk I've heard about 'curl | bash' is that web servers can detect whether the site is being accessed by a browser versus curl, and serve different content. Legit risky, good to educate about; don't curl | bash from sites you don't recognize. But this is a GitHub link!
I would agree that package managers are superior from a security angle, if most package managers had strong security review processes, on both the package content and the publisher. The problem is: they don't. They're all lacking in this regard, from apt to flatpak to the Apple App Store. The most sophisticated is probably the Play Store, but binary command line distribution channels are nowhere close to having reproducible builds, verified Real ID developers, static analysis on distributed binaries, etc. Package managers are literally just "curl | bash" with some steps added. By and large, anyone can publish anything under any name; educating people that they're "safe" is actually a massive disservice.
> The only legitimate risk I've heard about 'curl | bash' is that web servers can detect whether the site is being accessed by a browser versus curl, and serve different content
This is the reason why you shouldn't do curl ... | bash, you should do bash -c "$(curl ...)"
The former is truly vulnerable, for the exact reason you mention, while the latter isn't.
> With curl | bash, the URL of the bash script is right there. I can copy it, plug it into my browser, and inspect with ease the shell script that will run
More savvy users might have set their shell up in a way that mitigates this risk, and some users will, as you mentioned, copy and paste URL, but go ahead and try that with the command in the link I provided - note that if you copy just the git URL, it only copies that bit, and then you can paste it into your browser, and not realize that there's a malicious bit before it
Of course, copying and pasting a command involving apt/dnf/etc is also vulnerable to that same attack :)
'bash -c $(curl )' doesn't mitigate a website's ability to detect whether you're downloading a file via curl versus accessing it in a browser (e.g. if (headers.USER_AGENT.includes("curl")) {}).
It does mitigate a different issue whereby the website can detect if you're piping the output of 'curl' directly to bash.
The safest option is definitely --> curl to file, audit what you downloaded locally, not in a browser, then bash -c that downloaded file.
But, again, this only really mitigates attacks from sketchy websites. If the website is trustworthy; both the website you're getting the curl | bash download string from, and the website the script is hosted at, such as Github, where 90% of these scripts are hosted; its not really mitigating a legitimate threat.
The copy-paste thing is interesting, but frankly: That's an argument for why the web is insecure, and that insecurity impacts every single method of software distribution which touches it. Copy-pasting a 'sudo apt-get install' line is the same class of risk here as copy-pasting a 'curl | bash' or 'bash -c $(curl)' line; you could swap the script to a new URL, or you could have them install a malicious package from apt, or even just nuke the apt-get install part and just do a curl | bash anyway. Definitely something that's great to know and educate about, but its not the strongest argument against curl | bash.
Honestly, this kind of made my blood boil, and just points out how difficult security can be.
Here is presumably a security company, in their own blog post outlining a high-severity vulnerability alert, advocating that you run untrusted code from some rando GitHub repo. It also outlines my fear about dealing with some "security consultants", because on the one hand they outline all of the things you need to "box check" for some security audit, but at the same time leave you less secure because they've just opened another giant gaping attack surface for your company - this is exactly what happened with the SolarWinds attack.
While I understand the concern with curl | bash, this method is used in many different open source product installations. The sh file is coming from github - pretty trustful source. You can always watch it in the browser (github will not trick you into different version) but you can also curl it or just download the source from the repository (just half a page above).
You can also always review and rebuild the entire tool - another beauty of the open source.
But most people just want to use the tool as fast and as simple as it can be. I guess there are options for any possible taste.
The only fix so far is removing the feature.
>According to the publication, multi-tenant environments where non-admin users have permissions to create Ingress objects are most affected by this issue.
Ouch! I guess this means lots of secret/password rotations for k8s admins coming soon. I've always wondered if there are public Kubernetes multi-tenant setups in the wild rather than just ones with admins and developer roles/namespaces.