[adrtessier]

There sure are a lot of these systems these days. While I'm always happy to see innovation in this area, I'm personally beginning to get confused as to why I might prefer to use (and probably then contribute to) one of these projects over another. After all, there's this, and Lyft's Confidant [1], and Square's Keywhiz [2], and plenty more that don't come to mind right now. They all have nice documentation about what they do, but none of them sufficiently explain to me what architectural differences they have, their pros and cons. I think it would be great to see that added to these products' pages at best, or at least some guy write a blog post about it.

[1] https://lyft.github.io/confidant/ [2] https://square.github.io/keywhiz/

[sciurus]

> none of them sufficiently explain to me what architectural differences they have, their pros and cons.

Have you looked at https://vaultproject.io/intro/vs/index.html ?

[whisk3rs]

This page does not adequately address many of the practical questions that orgs with secrets should be asking.

Examples:

* Does it integrate with my existing infrastructure security policies? * Does it introduce a new set of principals for me to track and manage? * Are the configurations stored in a cleartext manner suitable for storing in version control? * Will the audit logs integrate with my existing audit log tooling? * Does it help or hinder continuous deployment? In other words, will my code be littered by conditionals to support non-prod jobs? * Can my devops team run this or does it require additional staffing? * How many layers is this adding between me and the security primitives offered by my platform (Windows AD, AWS IAM, etc) * Is it serverless, or does it require me to run additional servers? * How do I ensure continuity if the software is no longer actively developed? * Can it deploy secrets to dev and test environments in the same way? * What's the revocation story?

The answers to most of these questions for most of the tools that have made it to HN recently are not promising. For AWS the best choice seems to be using KMS directly and integrating your app with the declarative configuration tools AWS offers (IAM, CF, MFA).

Outside of AWS, Vault is the only one that I've seen which handles real-world end-to-end in a responsible way. For example, Vault is the only one that doesn't focus strictly on storing static values: Vault can integrate with services such as Postgres and SSH to dynamically provision time-limited credentials (https://vaultproject.io/docs/secrets/postgresql/index.html).

[adrtessier]

+1 to what whisk3rs said, as that's how I feel, as well.

For example using Vault v. Keywhiz [1] since I made that comparison earlier, this page feels like it's written to say "Vault does X. Keywhiz does too." That's not really what I'm looking for; I can diff the two projects' features in that way as well, but I want to know why it is that Vault is better, i.e., why is it that it exists, what sets it apart from other solutions I can use? Why should I invest my time and port my infrastructure into this project? Should I expect it to be around or abandoned in a year if there is no adoption?

I probably should have clarified this in my original post; sorry for the confusion.

[1] https://vaultproject.io/intro/vs/keywhiz.html

[mcpherrinm]

I'm one of the Keywhiz developers, and we've been considering writing such a page, though I've been hesitant to do so because I'm not totally familiar with the alternatives: I have no operational experience with Vault or alternatives, and haven't put a ton of thought into this comparison.

From my limited perspective: Keywhiz is a system to take secrets, stored in a database, decrypt them with an HSM backed key, and expose them as files on a server. Because the secrets are exposed as a fuse filesystem, they require zero special code in your applications. Most other secrets stores just provide you with an API that requires custom integration work. You could probably even modify keywhiz-fs to talk to a Vault server.

On the other hand, Keywhiz doesn't know anything about the secrets inside it, so there's not much out of the box for secrets revocation, and other operations like that. There's also minimal secrets generation: We just provide an API to handle that. The biggest sources of secrets for us are service-to-service TLS certs, generated from our deploy system, which just calls the automation API to put the certs it gets from our internal CA. Keywhiz has a plugin system for generating secrets, but it's quite simple compared to what Vault has.

Vault also has some operational decisions that seem scary to me. They have a scheme for "unsealing" a vault with a set of secrets an operator has to input. This seems quite likely to cause an outage, while defending against an attacker you've noticed but who hasn't managed to steal the secrets out of your unsealed vault. While operating, your vault is unsealed, so an attacker who gets root has your secrets in both Keywhiz and Vault, but a service restart causes an outage. If you could somehow (via physical access, perhaps) boot a malicious OS onto a Keywhiz server, and your HSM doesn't defend against that, you could read secrets. For Vault, you also need to steal the keys to unseal. But those are accessible to humans, so that's probably easy.

As for lifetime, we don't expect Keywhiz to go anywhere. It's a crucial part of running Square. We've been using it for years now (though only open-sourced it earlier this year), and will probably continue to exist as long as the company does.

I'm happy to answer any other questions, too.

[mitchellh]

I don't want to make a big comparison right now, so just addressing your specific concern about Vault:

Unsealing doesn't cause an outage since you run them in HA-mode (side by side, leader elected).

As you said, if you gain root, both in Keywhiz and Vault, you can extract secrets at will. I can't comment to Keywhiz's capability (I'm sure some exists), but Vault will send audit logs to multiple backends so at least you can see it happening. The root token in theory can disable audit backends, but that action itself is also audited before it is disabled. That scenario in particular (root token compromise) is the worst case scenario. Ideally you throw that root token away into a N-key area as soon as possible. This feature is coming to Vault soon (expanding the shamir to arbitrary paths/secrets in Vault to support N-key operations).

Vault itself is used in places with very very high requirements of uptime (one user in particular can measure their overall downtime in seconds in the last quarter century). So we've had to go through some rigorous due diligence in such places to make sure it can withstand that. So, I assure you operationally that it won't cause an outage if it is properly configured in HA-mode. :)

[Rapzid]

I hate to nit, but do the applications really not need special code? Most applications require secrets in one of their own config files, or through some other mechanism that is not some file in some format. If you need to write the code to get this secret off the file system and to where the app can use it.. What's the difference between using a simple client to fetch from vault vs fetch from the file system?

I don't currently use either vault or keywhiz(we rolled our own solution using KMS and dynamodb) so I'm not really knocking either method. The current state-of-the-art with configuration management and secret distribution(particularly for container stuff) still requires a lot of customization code specific to each wrapped application, unless you wrote it all yourself to suite.

At the risk of sounding like a fanboy though, I will say that my own solution and others that have come out feel very itch-scratchy compared to vault. From what I have read about vault it has a much more thought out and productized vibe. Lots of cool features beyond delivering an encrypted blob from point A to B. I get this impression from a lot of Hashicorp components in general, and feel it's a shame larger open source projects like Kubernetes aren't integrating them while letting scaling to 200 hosts be a blocker to their next release... But I digress.

[mcpherrinm]

Well, our #1 use case is TLS certs, and basically every app supports reading them from a file.

For config files with secrets, there's often a way to include another file, or sometimes we just drop the whole config file into Keywhiz.

Here's an example from the Keywhiz codebase, of a file that's loaded from disk, but in prod could be swapped for a properly secured secret https://github.com/square/keywhiz/blob/master/server/src/mai...

In development, you don't need to run Keywhiz, so you can just write a file on disk, so that's a nice advantage: Less things to depend on.

[wmf]

There's probably some strategy tax at work here since Hashicorp is trying to build a self-contained ecosystem of devops/microservices tools.