[scrollaway]

First reaction: Holy crap! They finally turned ParameterStore into a proper product!

Second reaction: Holy shit that's expensive [for what it does].

ParameterStore is free (minus the KMS component). The only value-add is secret rotation and that's not something that most of the time makes sense to use. [Edit: I'm not advocating for no rotation; see replies]

Edit: Had more time to think about it. Someone enlighten me: What's the difference between writing a rotation lambda for this new product, vs. writing a rotation lambda for ParameterStore that you then cron? The pricing really doesn't make sense.

[Dunedan]

My biggest criticism of Parameter Store is actually that it's free. Let me explain:

Because it's free they limit the requests per seconds you can make to the Parameter Store. That's especially noticeable when doing requests for all parameters of a given path, as the limit is way lower there than for requesting (a bunch of) individual parameters. In the past that caused serious problems for us when using Parameter Store for AWS Lambda functions during a deploy of new versions of functions, as suddenly there was a spike in the number of requests to Parameter Store as all AWS Lambda containers got replaced.

They of course set such limits because it's free, so I'd gladly pay for getting increased limits.

[scrollaway]

Oh yeah, I agree - and I have no beef with the per-request pricing. It's the per-secret pricing that's weird. If I understand this announcement correctly, you're still paying for KMS separately as well, so that's not where the price spike comes from...

[drodgers]

Oh, yeah, we just ran into this recently when rolling-out parameter store. For anyone else having this problem, the trick was to use the GetParametersByPath API method (https://docs.aws.amazon.com/systems-manager/latest/APIRefere...) instead of the more flexible DescribeParameters (which has stupidly low, undocumented rate limits).

[Dunedan]

At the time we experienced the problems GetParametersByPath had way lower limits than GetParameters (~an order of magnitude)! That's the problem we ran into. That was already back at the end of last year and the SSM team promised to improve the situation, but so far at least no publicly available news states something about it.

[toomuchtodo]

> The only value-add is secret rotation and that's not something that most of the time makes sense to use.

From a security perspective, you should be rotating secrets somewhere between annually and every 90 days, depending on your business/security/compliance requirements and the nature of the data secured by the secret.

[scrollaway]

You're not necessarily your own source of secret (and even when you are, you don't necessarily have the option to use AWS-sourced rotation).

In other words, yes, you should be rotating what you can rotate, but this doesn't always help. Furthermore, it doesn't justify the pricing. It would make sense if this were, say, "Hey, you can now auto-rotate SSM-PS secrets for an extra $0.40/secret".

Right now it just seems weird.

[Edit: I just saw the custom rotation bit of the article. Cool; but if you're at the point of setting up lambdas for the rotation, you might as well cron a lambda on top of ParameterStore...]

[mlrtime]

Does it matter? How many secrets do you have where this is even close to your ec2/storage costs?

[tptacek]

It depends on the secret and the degree to which the secret is exposed. SSH creds should get rotated constantly; a one-hour SSH login cred is a significant exposure. But an API secret that is kept in Parameter Store and not exposed to developers doesn't really benefit from rotation every 3 months in proportion to the amount of mechanism required to do that.

[JoachimSchipper]

What are you assuming about exposure? If a SSH key lives on a well-secured workstation or bastion host (and you ideally don't agent-forward it to insecure hosts), rotating that key once per hour doesn't seem a top priority to me? E.g. a sudo password is (lower-impact, but) more likely to get exposed to compromised hosts?

("Well-secured workstation" is arguably an oxymoron, of course...)

[cassianoleal]

Wait, you keep private SSH keys on bastion hosts?

It's much better practice to use the bastion as a proxy to the other hosts. This is easily achieved using the ProxyCommand option of OpenSSH.

No agent forwarding, no secrets kept in random hosts.

[JoachimSchipper]

No, keeping SSH keys on bastion hosts was the silliest reasonable thing I could think of.

[scrollaway]

Do you know a good guide with standard practices for setting up and securing bastion hosts (preferably on aws)?

[toomuchtodo]

If you don’t require interactive sessions, consider using AWS SSM run command [1] instead. You install the agent on the instances, with commands sent from the client through the AWS control plane (with IAM and SSM documents for access control and CloudTrail logs of all commands issued).

I’m currently deploying it in an enterprise for ~5k users, and it works surprisingly well for providing the ability to run arbitrary commands on instances without ssh access.

[1] https://docs.aws.amazon.com/systems-manager/latest/userguide...

[jessaustin]

...a one-hour SSH login cred is a significant exposure.

Does this refer to an SSH password or e.g. a 4096-bit RSA key, or possibly something completely different? If this refers to an RSA key then I'm quite confused.

[apawloski]

How are you managing your SSH keys to make <1HR rotation feasible?

[tptacek]

With an SSH CA.

[kasey_junk]

Is there one of those you’d recommend?

[skywhopper]

Check out Hashicorp Vault (https://www.vaultproject.io/docs/secrets/ssh/signed-ssh-cert...) for a general product (has lots of the features of AWS Secrets Manager as well), or Netflix BLESS (https://github.com/Netflix/bless) for a very specific tool for just this.

[euroclydon]

I found these:

https://www.digitalocean.com/community/tutorials/how-to-crea...

https://medium.com/uber-security-privacy/introducing-the-ube...

[gregmac]

What's the rationale behind rotating? I've never really understood this. I'm asking this from a fundamental point-of-view: I realize there are policies and standards that require rotation, but I want to know why those policies say that in the first place.

If you have a secret, it's either (potentially) compromised and should be changed immediately, or it's not compromised and thus doesn't need to be changed.

If you're not sure if it's compromised or not, you have an entirely different problem that isn't solved by rotating. Put another way, rotating every 90 days is the equivalent of saying "it's okay if this secret is compromised, because no one can do any damage with it if they only have under 90 days" -- and I can't imagine any situation in which that would be true or acceptable.

[AnimalMuppet]

You seem to be thinking of a detected compromise, which may have compromised secret X, and your response is to change secret X right now. And I think that's the correct response.

But in another sense, all your secrets are potentially compromised, all the time. That is, there are compromises that you don't detect, or don't detect right away. They happen. What should you do? Well, you change the secrets every 90 days, even if you don't (yet) know that they have been compromised, to minimize the damage of a compromise that you don't know about.

But, you may say, by that logic shouldn't we change them every day? Every hour? But the other tradeoff is with annoying your users, wasting peoples' time on changing secrets, wasting time as they have to try to remember today's secret, and all the associated hassle. It's a compromise between better security and better usability.

Is 90 days the right compromise? I don't know.

[gregmac]

If you're talking about secrets used by users (which isn't what AWS Secrets Manager is used for), that is specifically covered by NIST 800-63B [1] which recommends:

* Do not impose other composition rules (e.g. mixtures of different character types) on memorized secrets.

* Do not require that memorized secrets be changed arbitrarily (e.g., periodically) unless there is a user request or evidence of authenticator compromise. (See Section 5.1.1 for additional information).

There's research [2] backing the idea that these policies actually lower security because of the way users act in response.

Systems don't have these problems, so rotating every few seconds doesn't seem to be a big deal. However, if the secret was compromised and you don't know how, you have to assume that the new secret can also be compromised (via the same mechanism), and at the same time there's no reason to assume that an attacker can't make use of it before the next rotation. Thus the rotation is completely ineffective as a means against this unknown attack, so why bother? The energy would be better spent eliminating potential for unknown, undetected attacks.

[1] https://pages.nist.gov/800-63-3/sp800-63b.html#-1021-memoriz...

[2] https://www.ftc.gov/news-events/blogs/techftc/2016/03/time-r...

[AnimalMuppet]

> The energy would be better spent eliminating potential for unknown, undetected attacks.

Think both/and, not either/or. Yes, work on eliminating unknown, undetected attacks. But you're never going to be perfect at it, so also work on minimizing the damage from undetected attacks. If you're not dealing with human-remembered secrets, sure, go down to rotating every few seconds. By doing so, you make the compromise harder to exploit. Even though you don't make it impossible to exploit, "harder" is still worth doing.

[mrep]

Got a link for rotation time frame recommendations because I have seen them all over the board. I understand the basic one that you want old employees to have no knowledge of current credentials but I have worked with security rotation requirements for minutely, hourly, daily, monthly, and yearly and I've never really seen a reasoning for the difference between them.

[talawahdotnet]

The pricing is a bit surprising, but I am sure that if you tried to build the full solution you would end up needing a few more pieces than just the rotation lambda and cron.

Would have been nice if there was a 5 secret free tier though. I feel like the sticker shock will dissuade small users when everybody really should be doing secret rotation.

[niklaslindblad]

After Amazon EC2 Systems Manager Parameter Store got some much needed love last year (adding KMS encryption and versioning), I wrote a small utility for populating environment variables using a specific Parameter Store prefix: https://github.com/nlindblad/confidential

It can either write the environment variables out to a Docker compatible environment file or it can be used as a wrapper to run any executable/script with the environment pre-populated. The README contains various examples of those use cases.

The primary motivation was to have proper secrets management for my hobby projects that didn't rely on Ansible Vault, configuration files in S3 or having to use the AWS SDK in each project.

[mlrtime]

Aren't all your secrets available as environment variables on your instances? If someone has shell access or gets the environment access pragmatically, all your secrets are compromised.

[niklaslindblad]

True, but if someone gets shell access as your application user, there is nothing stopping them from impersonating your application or reading any configuration files (including AWS credentials) stored on the file system.

If you are using IAM roles, a HTTP call to http://169.254.169.254/latest/meta-data/iam/security-credent... would give any attacker with shell access the credentials needed to iterate through any secrets in the Parameter Store that the instance has been given access to.

For shell access as another (non-root) user that is not running your application, it is just as safe to rely on environment variables as having configuration files with proper file permissions: https://security.stackexchange.com/questions/14000/environme...

[donavanm]

Wrt rotation theres also a cultural component of it. Systems grow in complexity and importance. Forcing periodic rotation, With corresponding automagic client support, ensures thats its even possible when you need it. Compare STS tokens to IAM credentials.

[Florin_Andrei]

What's the backend? If it's HSM, that would explain the steep price (sort of).

[weej]

Would be very surprised if it's not backed by AWS HSM. About 2 years ago I believe AWS KMS-->HSM was leveraging SafeNet's Luna HSM.

[inopinatus]

The SafeNet product is now called "CloudHSM Classic". The current CloudHSM is documented to use the Cavium NITROX HSM, which in the PCI HSM format is partitionable into "virtual HSMs". This looks pretty much designed for cloud HSM providers.

Secrets Manager is very likely backed by KMS which appears to use a KMS-specific HSM module (judging by https://csrc.nist.gov/csrc/media/projects/cryptographic-modu...)

[staticassertion]

How many secrets do you have? Even with thousands this doesn't seem expensive, and if you have 1000's you're probably mature enough to pay a bit.

[scrollaway]

To be clear, this would be peanuts to my company. But it's super expensive for what it is especially because they have a free service that does the same thing right next to it.

[jcolella]

Totally had the same reaction. The first thing I thought of is "wait doesn't ParamStore already do this?"

[januszeal]

>The only value-add is secret rotation and that's not something that most of the time makes sense to use.

This attitude is why company after company keeps leaking customer data.

[scrollaway]

This hasty drawing of conclusions from a post that wasn't suggesting you shouldn't rotate secrets is why populist after populist keeps getting elected.

[Johnny555]

Is it really a "hasty drawing of conclusions" when it's based on a simple reading of what was written?

[secret rotation,] that's not something that most of the time makes sense to use.

[scrollaway]

I just had a quick glance at my own app. The majority of the secrets in it are generated outside of AWS and (unless I'm misunderstanding the announcement) can't be rotated by this service without custom logic to rotate them; at which point, as I said in a separate post, I might as well write that custom logic on top of ParameterStore and cron it.