[otterley]

How does SSH agent forwarding work if there's no interactive user driving the process? Otherwise the concurrency is going to be limited to however many ssh sessions the client can spawn; and the orchestration/deployment cannot be a handsfree process.

[ntucker]

I'm not the person you're asking, but in our system, our secret access is tied to an SSH identity, not an SSH agent. This means that you have flexibility to require a personal user's ssh agent to be present for access to some things, or an identity file (optionally encrypted) stored on the server for less-sensitive things. In the case of things that need to happen unattended, we use an in-the-clear identity file, which might seem a little counterintuitive since security-wise, it's basically equivalent to just having the secret sitting there, but the indirection adds ease of rotating secrets along with revocability if we were to suspect we lost control of such a machine. But for most of our secrets, we tie them to individual user's identities which must be forwarded via an agent. We consider this an acceptable constraint because deployment and app restarts are always driven by a human in our operations.

It's not a perfect system or a panacea, but it's far better than having a bunch of passwords directly stored in config files, in my opinion. We also have a wrapper tool that uses your SSH agent access to fetch secrets for the mysql suite of tools as needed, so nobody is ever tempted to create a personal .my.cnf on prod servers. Anybody who actually has access to a given mysql password is still on the honor system not to fetch it and write it down somewhere, but removing the need to ever even see them eliminates the temptation to put it in a .my.cnf or in your clipboard, which eliminates a whole class of mistakes you can make in handling them.