[yandie]

I used to work for AWS and had to deep dive into IAM to build a feature.

Basically Everytime you touch AWS your session is tagged with your credentials and has a unique ID. So everything downstream you touch has your session ID associated with it.

Now say somebody from Redshift wants to access the customer's data. They will then need to access to the encryption key in KMS. The trail will be there since KMS lives in the customer's account (you can audit your own access). And for production services, human actors cannot access these keys - only production credentials can. An engineer who can log into a prod host in theory can grab the temporary credentials there but it expires in 15 minutes so your trail will be rather visible. Also access to prod host has a high bar - only senior people can do it.

Now in theory somebody can coordinate with a malicious user in KMS team - but the bar is high. Also the actual master key never leaves the premise for KMS so your attack surface is very limited.

Of course there are some core teams like IAM and KMS where if they become vulnerable the whole thing falls apart. But that's a big stretch for those systems since they are the core to the business.

[jacquesm]

This is about as bad a revelation as the original one. So the encryption key is fair game without explicit customer approval?

[dodobirdlord]

I think perhaps you misunderstand the architecture of KMS. KMS master keys are used to remotely decrypt the symmetric encryption keys for encrypted data that are stored alongside the encrypted data. KMS master keys don't ever leave the KMS servers themselves, and servers can't be accessed directly by anyone. AFAIK they don't have open ports except for handling production traffic and are hardened against opening a shell. An engineer on a different team with access to a host running a customer workload could potentially run off with a temporary customer credential being used by the customer workload, which they could then use to call KMS to decrypt encryption tokens for as long as the credential lasted. But they couldn't get at the KMS key itself or retain access past the expiration of the stolen credential, and all of the aforementioned audit logs would report all of the activity of the stolen credential.

[jacquesm]

I think you misunderstand my concern. What I'm missing in the above scenario is that a resource that should be 100% under the control of the customer and nobody else can be accessed by AWS personnel to open up a door that should be closed unless the customer permits access.

What the technical implications are is moot, the process that hands out these credentials should not be accessible to anybody but the customer. It implies that AWS personnel can impersonate customer representatives or processes run on behalf of those customers. That's a serious problem.

In all the years that I've been co-locating I do not remember a single instance where a representative of the hosting facilities that I've used gained access to our data or hardware without my very explicit permission.

As for audit logs: they are only as useful as those inspecting them, and more often than not are entirely passive until required for evidentiary purposes.

[dodobirdlord]

> It implies that AWS personnel can impersonate customer representatives or processes run on behalf of those customers. That's a serious problem.

Rather than being a serious problem I think it's more on an obvious fact. AWS personnel build services that specifically exist to act on the customer's behalf with delegated credentials. Any time you configure a managed service to run with an IAM role, that service assumes the role and acts with the credentials granted to the role. AWS personnel have access for emergencies to the systems running their services, and by their very nature those services are in possession of customer credential sets for the IAM roles that the service is configured to use.

For example, a Lambda Function can be configured to run with a particular role. When the Lambda service goes to run the function, it fetches the role credentials from IAM and makes them available to the running Function. It could not be otherwise, because the purpose of a managed service like Lambda is to carry out actions on behalf of the customer. The role's credential set is as much a piece of data as the code of the function to be executed.

But leaving all of this aside, of course AWS personnel can access any and all data you store in their systems. They are legally obligated to turn whatever you have stored over to the courts in response to a warrant. So not only could they gather up your data by this roundabout method of misappropriating credential sets, they must have a way to simply access all of the data directly in a way that doesn't appear in audit trails. I assume for simplicity that the IAM service simply has an endpoint accessible to the company's lawyers that will serve up forged customer credentials on demand.

[donavanm]

I believe youre misunderstanding how KMS works and is exposed. You probably want to look at the concept of “kms grants.” Thoese regulate which principals, including service principals, can use CMK materials. The customer controls those grants. There are also substantial public docs, and more available on request, around the implementation, certification, and compliance of KMS infrastructure. If KMS is insufficient for your needs CloudHSM is availble for something even closer to “hosted HSM” than “key service.”

In short IAM controls everything, there is no “back door” or universal admin access, and KMS is used to perform sensitive operations NOT handing secrets to arbitrary (internal or external) consumers.

[throwawaybkhv]

some1 with the right access to the kms service could change a key policy to allow access to a bad guy. in theory. bcuz some1 has to have access to key policies since customers lock themselves out of their keys all the time.

but no 1 can export the private key itself. and key policy changes are vry heavily audited by aws (and can be by the customer, too). this is all proven by the 3rd party audits aws receives

[yandie]

Yes, they can. However, that will leave their trails in their KMS service CloudTrail - unless they manage to exploit CloudTrail as well. That's a lot of barrier to bypass, especially because accessing all these services require you to be in the correct permission group with a hardware MFA token.

Somebody can access the key hardware but they can't extract the actual key out of that. However, I've never met anyone with that level of access - and AFAIK you have to go through various security clearance and approval before such human intervention is permitted.

There's no such thing as perfect security - but KMS is as solid as I can see with centralized key management at the moment. And customer can roll out their own key server as well that is managed in your own data center.

[StopHammoTime]

Plus, if there is any legitimate concern about AWS having access to KMS keys (at this point it would be that they own the servers, and that's about it), you can roll a CloudHSM and import your own keys.

KMS is very clear about it's usage and what it involves. It's obvious that with Symmetrical Encryption AWS obviously needs to know the other end of the key at some point so that it can decrypt the data.

However, as customers can't even export these keys and the whole system is based on using KMS to actually perform the decrypt operations it is a non-starter. It's a lot more secure than most infrastructure which probably encrypts locally but is stored in a broom cupboard with a $10 lock.

[donavanm]

> It's obvious that with Symmetrical Encryption AWS obviously needs to know the other end of the key at some point so that it can decrypt the data.

Its worth noting that even symmetric keys dont imply direct access to the secret itself. You can instead use the highly controlled secret material to derive less sensitive material. For example a hash derived from a known input + the secret. A third party can use this to prove that two other parties both have/had access to the shared secret. But the third party never needs to access the secret itself.

Theres a great example of this in the chained hashes that make up an AWS sigv4 API request signature. https://docs.aws.amazon.com/general/latest/gr/sigv4-calculat...