[scrollaway]

What do people here use to store and source-control secrets/almost-secrets and make them available to (pick n) terraform/ansible/salt/chef/...?

I've heard a lot of good things of Hashicorp Vault (https://www.vaultproject.io) but been hesitant to go with it.

[ejcx]

I almost never heard anything negative about vault until I did a presentation on all the troubles I had with it. Here are the slides about the issues I had:

https://docs.google.com/presentation/d/1ipP2eB9pW5j3WDvzCGz9...

At the end of the day, Vault gives you secret keys to manage which makes automating the unsealing process not fun if you want full automation.

Talking with Armon from Hashicorp they planned to work on some much improved docs around vault which should help with a lot of the issues of making vault usable, because quite frankly they are very challenging to understand right now.

[lolbrish]

I unseal using ansible (with the unseal keys in ansible-vault) and automate the configuration fully through ansible. For example you can use the ansible expect module:

  - name: unseal 1
    expect:
      command: '/usr/bin/vault unseal'
      responses:
        'Key \(will be hidden\): ': "{{vault_seal_key_1}}"
      echo: yes
    when: vault_sealed_result.rc == 2 and vault_seal_key_1 is defined
    tags:
      unseal

[bogomipz]

I'm interested in your solution, you are using ansible-vault to store the Hashicorp Vault unseal key(s)? Isn't this just pushing the problem out another level or am I missing something? Thanks.

[jaxxstorm]

I wrote a tool to ease this: https://github.com/jaxxstorm/unseal it's not perfect, but it makes the process much, much easier

[redwolfb14]

I read this presentation a little while ago and I felt it was slightly disingenuous. Many of the "problems" are process related and have nothing to do with vault itself. Then your approach is using ParameterStore which provides a UI and is built on the AWS ecosystem proper which I guess works if you are sticking to only AWS. I guess it comes down to how you view "secrets" but i'd rather that be spelled out than suggesting hard to automate and not easy to work with.

[bogomipz]

Thanks for sharing. Was your talk recorded by any chance? I would be interested in watching it.

[scrollaway]

aws-vault and chamber both look fantastic. When was the talk given? Has the situation improved since? Would you still recommend both those tools over Hashicorp Vault?

[Johnny555]

Another option for Ansible is Ansible Vault (which is not related to Hashicorp Vault) -- you can use it to password protect secrets used for playbooks (you need to supply the password when you run the playbook).

https://docs.ansible.com/ansible/latest/playbooks_vault.html

[toomuchtodo]

Ansible Vault is the bees knees compared to Hashicorp Vault (if you're already using Ansible). Very pleasant to use, A++ would use again.

[bogomipz]

Yes but Hashicporp Vault has a greater scope in that it addresses the secure introduction problem, provides single usage/read once tokens, tokens with TTLs and limited use, has audit capabilities etc. Although there is some overlap in general secret management between Ansible Vault and Hashicorp Vault, the latter is much broader than just a means of secure storage for config management.

[toomuchtodo]

A lot of what Hashicorp Vault does is already provided for (in AWS at least) with KMS, Cloudtrail, Parameter Store, and IAM (which can be used in symphony with Ansible Vault).

I have very little experience with GCP and Azure, but it seems like Hashicorp is reinventing the wheel in AWS with Vault.

[bamboozled]

Same, love it !

[thinkmassive]

> you need to supply the password when you run the playbook

You can also specify vault-password-file in ansible.cfg [1] It can be a shell script rather than plaintext, so you can use it to call the CLI password manager "pass"[2] for instance. This is handy for automation.

[1] http://docs.ansible.com/ansible/latest/intro_configuration.h... [2] https://www.passwordstore.org/

[ams6110]

Of course if you're putting the encryption password in ansible.cfg you might as well just leave the original file unencrypted.

[linsomniac]

The parent's recommendation was to put a filename into the ansible.cfg, and that file could contain either the password, or a script that is then run which prints the password to stdout. For example, we have the script pull the password from a gpg-encrypted file.

[finnh]

That's not what GP said. "vault-password-file" != "vault-password".

[Johnny555]

Even if you do put a plaintext password into ansible.cfg, encrypting secrets in the playbook is still worthwhile so you don't commit them to your source code repository or accidentally share the secrets with the world when you publish your playbooks.

[linsomniac]

We also use Ansible Vaults extensively and they work great.

Oddly enough though, I recently tried AWX (the open sourced Ansible Tower), and it wouldn't decrypt our vaults when trying to get the inventory, even though I gave it vault credentials (there was nowhere to associate it with the inventory run though).

So, we are still using RunDeck for a web UI/scheduling/web triggers/Slack integration of our Ansible runs.

[matburt]

developer of AWX here. We're working on this! unvaulting is available during playbook runs but we definitely need to make it available during inventory syncs also. The features coming up in Ansible 2.4 will enable us to do this.

[linsomniac]

Are there any pointers on how to work around this?

I think a large part of our problem is that we are using Ganeti for most of our VMs, rather than something supported native by AWX like OpenStack/EC2/Azure. I have an inventory script we have been using, but couldn't get it to run in AWX due to AWS credentials not being made available in a way that boto recognized (the inventory is pulled both from Ganeti and EC2).

[corford]

Another happy Ansible Vault user here. It's simple and effective.

[lmickh]

Can't recommend Vault enough. By far the easiest and most capable solution to work with. The only downside I can point out is that the multi-cluster/region HA requires expensive enterprise licensing, but that is something most user cases don't require.

[peterwwillis]

> By far the easiest

Secret management is as complex as the system which relies upon it. Vault is not as easy as many other tools designed for simple systems. GPG by itself is often enough to manage secrets.

There are probably 40 or more secret management solutions out there, many tailored for specific uses. Most CMS's have their own secret management baked in. Most orchestration and infrastructure tools do too. Four different solutions are called 'Vault'. There are at least 5 solutions just for Amazon. Depending on your platform, something other than HashiVault may be easier to adopt.

Here's some reviews of a dozen different solutions: https://gist.github.com/maxvt/bb49a6c7243163b8120625fc8ae3f3... https://medium.com/on-docker/secrets-and-lie-abilities-the-s... https://www.threatstack.com/blog/cloud-security-best-practic... https://www.youtube.com/watch?v=OUSvv2maMYI&feature=youtu.be

[lmickh]

Maybe that was poor wording on my part, but ease of use in combination with the features are important.

Take the database secret backend for example. Getting that same feature out of other simple systems, would be a lot of work. Audit trails are another low effort high reward feature. When you start to get into combining those features, it is an even greater pay off. When you start to take into consideration HA clusters...well if you want to put that together on your own have at it.

Much of "easiest" has to do with familiarity as well. I've seen new users get Vault up in minutes that still don't have GPG setup "cause it is hard". If you are already working with DynamoDB or Consul, you already know how to setup the storage. Those are common skills. I'm sure there are folks that fall on the other side of that use GPG often, but it doesn't make either side more objectively easy.

If you only have the need for a simple system, then Vault may be overkill. I would say Vault is comparable to Kubernetes. Does everyone need it: No. Simpler stuff can be done with config management or tools like Nomad. Does it have features that most people will eventually want to use: Absolutely.

Secret management is one of those things were simple solutions end up easily becoming more and more involved just like container orchestration. Especially when you get into chicken/egg scenarios regarding stuff like some of the tools you mentioned.

Side note: I've tried at least 12 other tools for this purpose , and I would recommend Vault over all of them for most every scenario that is more involved then "Use 1Password".

[odiroot]

Did you use it with Kubernetes by any chance?

[lmickh]

Yes. I use Terraform to pull secrets from Vault and insert them into Kubernetes secrets so that there is a single source of truth.

[puzzle]

What's the Enterprise pricing?

[lmickh]

IIRC, it was on the order of $100k+ per cluster for the Premium offering. Has been several months so I would suggest contacting them to see what it currently is.

For some projects I've been on, that wouldn't be a big deal. For smaller projects it can easily be several times your opex though.

You can run an open source cluster in HA without it though. The big draw for Enterprise is the multi-cluster and HSM support. Anything HSM is typically quite expensive so their pricing really isn't out of line.

[hilljgo]

Here's a great blog post about how we use it here at Fleetsmith (disclosure - I work there)

http://blog.fleetsmith.com/authenticating-to-hashicorp-vault...

[rraval]

We use StackExchange's blackbox [1] to check in GPG encrypted secrets into our monorepo (contains both ansible deployment playbooks and the code itself). This way, we can introduce secrets and the code that uses it in a single commit!

[1] https://github.com/StackExchange/blackbox

[kaishiro]

Ah, wish I would have seen this before commenting upstream. Will definitely give this a look.

[simplehuman]

+1 for blackbox. Very nicely done!

[sova]

Originally part of Puppet - those guys are masterful

[hamax]

If you want something very simple, we've written our own wrapper around aws kms - for go and python.

https://github.com/Zemanta/go-secretcrypt

[wahnfrieden]

There are a handful of other wrappers around AWS KMS to solve this problem - it seems like the best approach if you're already on AWS, no additional major infrastructure to manage. Thanks for sharing.

[lylejohnson]

I'm currently taking advantages of SaltStack's GPG renderer to store secrets in our version-controlled Salt code (see https://docs.saltstack.com/en/latest/ref/renderers/all/salt....).

[mcx]

We're on AWS so we use SSM Parameter Store. Works nicely with KMS and IAM.

[weej]

+1 Same. Automation and integration is fairly frictionless if you're on AWS.

[tetha]

Vault is a pretty good thing. For some time, we used chef attributes and environments, but by now we have migrate most stuff into vault. The migration was easy, because chef-attributes form a big JSON object, so you can translate that into a secret tree in vault in a straightforward way.

The worst part about vault is managing access in a secure way and granting access to the right parts imo. We leverage the pki backend (or rather, three dozen pki backends) to map nodes to their respective policies and that required quite a bit of tooling to make work. But now it does, it's secure, and if need comes, it should be easy to revoke secret access for a cluster.

[gobengo]

Hesitant why? It's pretty darned good. Kubernetes also has a Secret abstraction, but you probably don't want to start setting up Kubernetes just for secret storage. Vault is good at that.

[jaxxstorm]

kubernetes "secrets" are base64 encoded only, anyone with access to the resource can view the original secret.

[jesseendahl]

The lack of encryption at rest for Kubernetes secrets was one of the (many) factors in why we originally chose Vault.

That said, there's been a lot progress recently in this area recently. Starting in Kubernetes 1.7, you can optionally encrypt etcd at rest: https://kubernetes.io/docs/tasks/administer-cluster/encrypt-...

You also have a few good choices for the crypto. Two of the choices are Secret Box (XSalsa20 + Poly1305) and AES-GCM with random nonce.

Full list of providers, including info on strength + other considerations:

https://kubernetes.io/docs/tasks/administer-cluster/encrypt-...

[kaishiro]

Responding not so much with a solution but with a question. We currently encrypt files in place with gpg (secretKeys.json => secretKeys.json.encrypted) and have the source file (secretKeys.json) git ignored while the derivative file is just added to the repo.

This is admittedly a bit low tech, but could someone more well versed than me tell me what's wrong with this setup.

[czbond]

Then as part of your code or deployment, your private key is on a server and some process decrypts it? It is low tech, but a good solution. Only concern would be how that private key is stored, and if it could be compromised. (edited)

[kaishiro]

Thanks czbond. And yeah, that's precisely the setup. Appreciate the response.

[godzillabrennus]

Looks really powerful.

What kind of secrets do others recommend relying on third parties like this to manage?

[tuxxy]

Self promotion warning:

I work at NuCypher (YC S16), and we're doing some pretty interesting stuff on the Ethereum block chain to build a decentralized KMS using proxy re-encryption.

[tombert]

I've not personally used Vault, but Hashicorp's other products like Consul and Nomad are solid pieces of technology, so I would guess Vault is too.

[zimbatm]

git-crypt[1] is quite handy in cases where the credentials have to live along the code. It's a nice improvement over having credentials in clear in a git repo and relatively easy to implement.

[1]: https://www.agwa.name/projects/git-crypt/