[toomuchtodo]

> The only value-add is secret rotation and that's not something that most of the time makes sense to use.

From a security perspective, you should be rotating secrets somewhere between annually and every 90 days, depending on your business/security/compliance requirements and the nature of the data secured by the secret.

[scrollaway]

You're not necessarily your own source of secret (and even when you are, you don't necessarily have the option to use AWS-sourced rotation).

In other words, yes, you should be rotating what you can rotate, but this doesn't always help. Furthermore, it doesn't justify the pricing. It would make sense if this were, say, "Hey, you can now auto-rotate SSM-PS secrets for an extra $0.40/secret".

Right now it just seems weird.

[Edit: I just saw the custom rotation bit of the article. Cool; but if you're at the point of setting up lambdas for the rotation, you might as well cron a lambda on top of ParameterStore...]

[mlrtime]

Does it matter? How many secrets do you have where this is even close to your ec2/storage costs?

[tptacek]

It depends on the secret and the degree to which the secret is exposed. SSH creds should get rotated constantly; a one-hour SSH login cred is a significant exposure. But an API secret that is kept in Parameter Store and not exposed to developers doesn't really benefit from rotation every 3 months in proportion to the amount of mechanism required to do that.

[JoachimSchipper]

What are you assuming about exposure? If a SSH key lives on a well-secured workstation or bastion host (and you ideally don't agent-forward it to insecure hosts), rotating that key once per hour doesn't seem a top priority to me? E.g. a sudo password is (lower-impact, but) more likely to get exposed to compromised hosts?

("Well-secured workstation" is arguably an oxymoron, of course...)

[cassianoleal]

Wait, you keep private SSH keys on bastion hosts?

It's much better practice to use the bastion as a proxy to the other hosts. This is easily achieved using the ProxyCommand option of OpenSSH.

No agent forwarding, no secrets kept in random hosts.

[JoachimSchipper]

No, keeping SSH keys on bastion hosts was the silliest reasonable thing I could think of.

[scrollaway]

Do you know a good guide with standard practices for setting up and securing bastion hosts (preferably on aws)?

[toomuchtodo]

If you don’t require interactive sessions, consider using AWS SSM run command [1] instead. You install the agent on the instances, with commands sent from the client through the AWS control plane (with IAM and SSM documents for access control and CloudTrail logs of all commands issued).

I’m currently deploying it in an enterprise for ~5k users, and it works surprisingly well for providing the ability to run arbitrary commands on instances without ssh access.

[1] https://docs.aws.amazon.com/systems-manager/latest/userguide...

[jessaustin]

...a one-hour SSH login cred is a significant exposure.

Does this refer to an SSH password or e.g. a 4096-bit RSA key, or possibly something completely different? If this refers to an RSA key then I'm quite confused.

[apawloski]

How are you managing your SSH keys to make <1HR rotation feasible?

[tptacek]

With an SSH CA.

[kasey_junk]

Is there one of those you’d recommend?

[skywhopper]

Check out Hashicorp Vault (https://www.vaultproject.io/docs/secrets/ssh/signed-ssh-cert...) for a general product (has lots of the features of AWS Secrets Manager as well), or Netflix BLESS (https://github.com/Netflix/bless) for a very specific tool for just this.

[euroclydon]

I found these:

https://www.digitalocean.com/community/tutorials/how-to-crea...

https://medium.com/uber-security-privacy/introducing-the-ube...

[gregmac]

What's the rationale behind rotating? I've never really understood this. I'm asking this from a fundamental point-of-view: I realize there are policies and standards that require rotation, but I want to know why those policies say that in the first place.

If you have a secret, it's either (potentially) compromised and should be changed immediately, or it's not compromised and thus doesn't need to be changed.

If you're not sure if it's compromised or not, you have an entirely different problem that isn't solved by rotating. Put another way, rotating every 90 days is the equivalent of saying "it's okay if this secret is compromised, because no one can do any damage with it if they only have under 90 days" -- and I can't imagine any situation in which that would be true or acceptable.

[AnimalMuppet]

You seem to be thinking of a detected compromise, which may have compromised secret X, and your response is to change secret X right now. And I think that's the correct response.

But in another sense, all your secrets are potentially compromised, all the time. That is, there are compromises that you don't detect, or don't detect right away. They happen. What should you do? Well, you change the secrets every 90 days, even if you don't (yet) know that they have been compromised, to minimize the damage of a compromise that you don't know about.

But, you may say, by that logic shouldn't we change them every day? Every hour? But the other tradeoff is with annoying your users, wasting peoples' time on changing secrets, wasting time as they have to try to remember today's secret, and all the associated hassle. It's a compromise between better security and better usability.

Is 90 days the right compromise? I don't know.

[gregmac]

If you're talking about secrets used by users (which isn't what AWS Secrets Manager is used for), that is specifically covered by NIST 800-63B [1] which recommends:

* Do not impose other composition rules (e.g. mixtures of different character types) on memorized secrets.

* Do not require that memorized secrets be changed arbitrarily (e.g., periodically) unless there is a user request or evidence of authenticator compromise. (See Section 5.1.1 for additional information).

There's research [2] backing the idea that these policies actually lower security because of the way users act in response.

Systems don't have these problems, so rotating every few seconds doesn't seem to be a big deal. However, if the secret was compromised and you don't know how, you have to assume that the new secret can also be compromised (via the same mechanism), and at the same time there's no reason to assume that an attacker can't make use of it before the next rotation. Thus the rotation is completely ineffective as a means against this unknown attack, so why bother? The energy would be better spent eliminating potential for unknown, undetected attacks.

[1] https://pages.nist.gov/800-63-3/sp800-63b.html#-1021-memoriz...

[2] https://www.ftc.gov/news-events/blogs/techftc/2016/03/time-r...

[AnimalMuppet]

> The energy would be better spent eliminating potential for unknown, undetected attacks.

Think both/and, not either/or. Yes, work on eliminating unknown, undetected attacks. But you're never going to be perfect at it, so also work on minimizing the damage from undetected attacks. If you're not dealing with human-remembered secrets, sure, go down to rotating every few seconds. By doing so, you make the compromise harder to exploit. Even though you don't make it impossible to exploit, "harder" is still worth doing.

[mrep]

Got a link for rotation time frame recommendations because I have seen them all over the board. I understand the basic one that you want old employees to have no knowledge of current credentials but I have worked with security rotation requirements for minutely, hourly, daily, monthly, and yearly and I've never really seen a reasoning for the difference between them.