[JohnMakin]

I was an extremely early user and owner of a very large-scale Vault deployment on Kubernetes. Worked with a few of their sales engineers closely on it - was always told early on that although they supported vault on kubernetes via a helm chart, they did not recommend using it on anything but EC2 instances (because of "security" which never really made sense their reasoning). During every meeting and conference I'd ask about Kubernetes support, gave many suggestions, feedback, showed the problems we encountered - don't know if the rep was blowing smoke up my ass but a few times he told me that we were doing things they hadn't thought of yet.

Fast forward several years, I saw a little while ago that they don't recommend the only method of vault running on EC2, fully support kubernetes, and I saw several of my ideas/feedback listed almost verbatim in the documentation I saw (note, I am not accusing them of plagiarism - these were very obvious complaints that I'm sure I wasn't the only one raising after a while).

It always surprised me how these conversations went. "Well we don't really recommend kubernetes so we won't support (feature)."

Me: "Well the majority of your customers will want to use it this way, so....."

Just was a very frustrating process, and a frustrating product - I love what it does, but there are an unbelievable amount of footguns laden in the enterprise version, not to mention it has a way of worming itself irrevocably into your infrastructure, and due to extremely weird/obfuscated pricing models I'm fairly certain people are waking up to surprise bills nowadays. They also rug pulled some OSS features, particularly MFA login, which kind of pissed me off. The product (in my view) is pretty much worthless to a company without that.

[neurostimulant]

They probably don't want their customers to use a competitor's product instead of Nomad.

[kensey]

> was always told early on that although they supported vault on kubernetes via a helm chart, they did not recommend using it on anything but EC2 instances (because of "security" which never really made sense their reasoning).

The reasoning is basically that there are some security and isolation guarantees you don't get in Kubernetes that you do get on bare metal or (to a somewhat lesser extent) in VMs.

In particular for Kubernetes, Vault wants to run as a non-root user and set the IPC_LOCK capability when it starts to prevent its memory from being swapped to disk. While in Docker you can directly enable this by adding capabilities when you launch the container, Kubernetes has an issue because of the way it handles non-root container users specified in a pod manifest, detailed in a (long-dormant) KEP: https://github.com/kubernetes/enhancements/blob/master/keps/... (tl;dr: Kubernetes runs the container process as root, with the specified capabilities added, but then switches it to the non-root UID, which causes the explicitly-added capabilities to be dropped).

You can work around this by rebuilding the container and setting the capability directly on the binary, but the upstream build of the binary and the one in the container image don't come with that set (because the user should set it at runtime if running the container image directly, and the systemd unit sets it via systemd if running as a systemd service, so there's no need to do that except for working around Kubernetes' ambient-capability issue).

> It always surprised me how these conversations went. "Well we don't really recommend kubernetes so we won't support (feature)."

> Me: "Well the majority of your customers will want to use it this way, so....."

Ha, I had a similar conversation internally in the early days of Boundary. Something like "Hey, if I run Boundary in Kubernetes, X won't work because Y." And the initial response was "Why would you want to run Boundary in Kubernetes?" The Boundary team came around pretty quick though, and Kubernetes ended up being one of the flagship use cases for it.

[JohnMakin]

Thanks for the detailed explanation - some of what you say sounds familiar, but this was nearly 5 years ago so my fuzzy recollection of their reasoning - I recall it being something like they didn't trust etcd being compromised on kubernetes. My counterargument to that internally was "if your etcd cluster is compromised by a threat actor you have way bigger problems to worry about than secrets"

[kensey]

My vague recollection is that that concern was that the etcd store (specifically the keys pertaining to the Vault pod spec) could be modified in some way that would compromise the security of the encrypted Vault store when a Vault pod was restarted. It's been a long time since I remember that being a live concern though, so I've mostly recycled those neurons...

[JojoFatsani]

(I have no idea what your infra is so don’t take this as prescriptive)

My feeling is that for the average company operating in a (single) cloud, there’s no reason to use vault when you can just used AWS Secret Manager or the equivalent in azure or GCE and not have to worry about fucking Etcd quorums and so forth. Just make simple api calls with the IAM creds you already have.

[leetrout]

Caveat: the HCP hosted vault is reasonably priced and works well.

However, strong agree on using your home cloud's service.

We used Vault with Heroku and were happy.

[candiddevmike]

> Caveat: the HCP hosted vault is reasonably priced and works well.

HCP hosted Vault starts at ~$1200/month, you'd have to use a metric shit ton of secrets in AWS or GCP to come close to that amount. Yes Vault does more than just secrets, but claiming anything HC sells as reasonably priced is a reach.

[leetrout]

Ah, they have changed the public pricing page. Maybe we were on a grandfathered in deal. They had a starter package between free and enterprise with configurable cluster options that was $60ish a month. We heavily used the policies, certs and organization features that made it a no brainer for that price point for things outside AWS like Heroku.

We were running about $12/mo in aws secrets with no caching and no usage outside our aws services. I taught the team how to cache the secrets in the lambda function and it dropped to a buck a month or less.

If they killed off the starter package then you are right, there are only outrageous options and HCP would not be worth considering for small orgs.

[elzbardico]

This^ Unless you're a hybrid/multiple cloud environment, there's no much point in using Vault.

[JohnMakin]

ime that’s a way better product to use for secrets management unless you’re trying to do very advanced CA stuff.