[zhoutong]

We've had the same issues at CoinJar, and we used SSH agent forwarding to solve it. This is how it can work:

Every new instance of application server is provisioned automatically by a trusted server (which holds the key to a credentials server). During orchestration/deployment, the application server has a temporary permission to fetch the secrets it needed, through SSH agent forwarding. Once the deployment is done, the session will end and the application server can never read new secrets until the next deployment.

This way we don't really need a solution like Vault. SSH is mature enough to provide authentication, and encryption is easy once you've figured out how to distribute keys automatically.

[ntucker]

This is similar to what we do at my company. I wrote about it a bit here: https://news.ycombinator.com/item?id=8826300

Would be interesting to hear more details of yours for comparison.

[otterley]

How does SSH agent forwarding work if there's no interactive user driving the process? Otherwise the concurrency is going to be limited to however many ssh sessions the client can spawn; and the orchestration/deployment cannot be a handsfree process.

[ntucker]

I'm not the person you're asking, but in our system, our secret access is tied to an SSH identity, not an SSH agent. This means that you have flexibility to require a personal user's ssh agent to be present for access to some things, or an identity file (optionally encrypted) stored on the server for less-sensitive things. In the case of things that need to happen unattended, we use an in-the-clear identity file, which might seem a little counterintuitive since security-wise, it's basically equivalent to just having the secret sitting there, but the indirection adds ease of rotating secrets along with revocability if we were to suspect we lost control of such a machine. But for most of our secrets, we tie them to individual user's identities which must be forwarded via an agent. We consider this an acceptable constraint because deployment and app restarts are always driven by a human in our operations.

It's not a perfect system or a panacea, but it's far better than having a bunch of passwords directly stored in config files, in my opinion. We also have a wrapper tool that uses your SSH agent access to fetch secrets for the mysql suite of tools as needed, so nobody is ever tempted to create a personal .my.cnf on prod servers. Anybody who actually has access to a given mysql password is still on the honor system not to fetch it and write it down somewhere, but removing the need to ever even see them eliminates the temptation to put it in a .my.cnf or in your clipboard, which eliminates a whole class of mistakes you can make in handling them.