[sandeepkd]

While I agree that it should not have happened, at the same time its probably true that most people are never formally trained on security.

The real story here is a big gap in existing implementations where shared credentials are needed and used pretty much across all the systems but there are no good solutions for managing such use cases. People are naturally more sensitive about their personal secrets than something thats shared across the company/group

[mikestew]

The real story here is a big gap in existing implementations where shared credentials are needed and used pretty much across all the systems but there are no good solutions for managing such use cases.

This strikes me as so wrong, I wonder if I’m misreading your comment. For instance, team password managers are a thing. And IT teams at many large corporations are not passing around an unsecured CSV files full of passwords.

[sandeepkd]

Lets take a concrete example, suppose you have AWS root account credentials. Are you going to assign them to one individual identity or as a company you would keep them accessible to a group of admins. Its going to be the second choice almost for every big company which makes them shared credentials.

Coming to team password managers at high level, its a shared location guarded behind closed doors (probably encryption at transit and rest). They would be another set of software that every company specially small business or contractors may not be incentivized to pay for. Some one in their naivety considered Github as a safe enough place, assuming that the access is guarded which turned out to be wrong and exposed this thing.

Lastly IT teams in large corporations being secure is a myth for most part. Your root keys for the most popular CA providers were shared in plain text emails not so long ago.

[acdha]

> Lets take a concrete example, suppose you have AWS root account credentials. Are you going to assign them to one individual identity or as a company you would keep them accessible to a group of admins.

You’d use AWS Organizations so each admin authenticates using their own credentials, gets short-term credentials to access the member account for the handful of operations needing root, and audit usage. It’s not only more secure, it’s also easier:

https://docs.aws.amazon.com/IAM/latest/UserGuide/id_root-ena...

Old school, you’d have a shared password in an encrypted team vault (possibly requiring x of y users to decrypt it) and two FIDO tokens locked in a safe. Again, this is rare and at a federal agency you have a physical security team with 24x7 staffing so you can say “in an emergency, one of the people on this list can get a key out of a safe in the CIO’s office”.

[sandeepkd]

great, now apply this to a 4 person startup who are just focussed to get business somehow. This is not on their radar and they would not be willing to spend money to address this either cause its not a problem that they are even aware of.

This is a tip of ice-berg, companies like openai, anthropic, perplexity, stripe, all of them have implemented their authentication and security flows in some interpreted language (python, ruby, typescript) cause that was the readily available talent on their product teams and most likely a good number of them do not even have their dependencies locked in.

[acdha]

That’s a pretty different scenario than we’re taking here, but it still doesn’t salvage your previous comment. Those people could still use one of the password managers which support this, which again would be easier than what this guy did.

[sandeepkd]

I am not trying to find an excuse when something is clearly wrong, what I am trying to share is how we ended up in a particular situation. The scenario is not much different, the rationale is that security (secure practices) are not the part of the product offering for most products/contracts. I have lost quite a few battles to management for security, it does helps me to understand how people think and priortize. People don't care for what they do not understand.

[antonvs]

This organization is using AWS apparently. They would store the root account credentials in AWS Secret Manager. That costs $0.40 per month. People in the relevant admin group would have access to them. They would log in with their individual AWS credentials in order to access the root credentials if they need that.

But, requiring AWS root credentials itself is an anti-pattern and implies an immature organization. That should not be needed for day-to-day operation.

This is all just ignorance and incompetence, nothing more.

> Lastly IT teams in large corporations being secure is a myth for most part.

This is CISA. The Cybersecurity and Infrastructure Security Agency for the United States. Security is what they're supposed to specialize in.

The only potential excuse here is that DOGE gutted them to a point that has completely compromised their capabilities. However, this situation is bad enough that it suggests that problems predated that incident.

[sandeepkd]

To be honest I do not know how to respond to this, cause this plays out quite often this way and sounds pretty convincing on surface. Unfortunately this is the gap between theory and implementation. There is a reason why the ROOT credentials are called ROOT. In case of anything going wrong, all your regular user accounts would be locked, see how you lock yourself out of this circular dependency. ONE SHOULD NEVER NOT PUT THEIR ROOT CREDENTIALS IN THE SECRET MANAGER OF SAME ACCOUNT. Its a classical circular problem, compilers compiler type. For AWS itself they have this additional concept of management account that allows you to defer this problem to just one more level.

Bottomline, you can have any number of boxes to lock other boxes and put their key to bounding box, ultimately there would be one outermost box that is locked by key which is not in any box

[antonvs]

> In case of anything going wrong, all your regular user accounts would be locked

You're talking about a very specific and rare scenario, and certainly not something that justifies storing all your passwords in plaintext in a CSV file.

In almost all scenarios where you would need root credentials, having them in the provider's secret manager is fine.

Obviously you need to store root credentials outside of the secret manager as well, but that should be a "break glass" scenario that's only used in emergencies. And you don't store them in plaintext CSV.

> Unfortunately this is the gap between theory and implementation.

I don't disagree that there are many, many organizations that practice bad security. But that doesn't mean there are none that have good security. And one would expect CISA to have good security, otherwise there's really no point in its existence.

There's a difference between saying "this is what most organizations are like" and "this is the way it has to be". The former is true, the latter is false.

[Hikikomori]

We deleted the root credentials efter initial setup where we added mgmt iam accounts used by our automation. If we ever needed them we used the recovery process. All users and services use temporary credentials.

[sandeepkd]

I made an assumption that you have federated AWS account setup. One organization management AWS account and then federated accounts under it and you are referring to deletion of deletion of ROOT credentials in the federated accounts.

Considering thats not the case, what you just did is move the goal post to a account recovery process. Question becomes who has ability to recover the account, in case its tied with email then most likely it has to be a shared email box. What you have now is a much more fragile system in case of custom domains, where whoever is controlling the email domain (DNS management capability) can take over the AWS accounts.

[Hikikomori]

One account, org, federated, whatever. You don't need to store the root credentials.

An email per account where only security team has access. Whoever can modify domain can already do this.

[sandeepkd]

This would be a incorrect representation/comparison of the problem being discussed. The semantics of ROOT account changes in the case when a separate management IAM account is introduced. In this case the question would become how you are securing the ROOT credentials for the separate AWS IAM management account/tenant.

[Hikikomori]

What part of we store no root credentials is confusing?

[realo]

You are right... Most use Excel files ...

[throwawaypath]

>For instance, team password managers are a thing. And IT teams at many large corporations are not passing around an unsecured CSV files full of passwords.

It's CURRENTYEAR. No one should be using team password managers or files to store credentials. There should not be storable credentials.

[acdha]

None of this is true at the federal level, or at least wasn’t before the current administration. There are standards for all of this, and if you haven’t read them most are quite reasonable — I keep the NIST 800-63 reference handy anytime someone tries to say password expirations are a good idea — and there are people who are paid full time to enforce them.

Having a password list or static AWS credentials is not only a direct policy violation but also implies a number of other failures, from monitoring GitHub repo administration and secret scanning to failure to enforce policies against sharing credentials (part of everyone’s standard training), require use of phishing-proof authentication, failure to use short-term credentials, etc. One mistake can be an individual but this is a multiple-manager failure going up to the executive level.

[antonvs]

> shared credentials are needed and used pretty much across all the systems but there are no good solutions for managing such use cases.

What do you mean by this? There are password managers and more enterprise-oriented secrets managers, and application platforms typically have integration with them. Individuals shouldn't be using shared secrets. This is a completely solved problem and it's not difficult to set up properly, especially in a cloud environment like AWS, where you can use services like AWS Secrets Manager.

[Forgeties79]

> While I agree that it should not have happened, at the same time its probably true that most people are never formally trained on security.

This isn’t a grocery store or something it’s CISA. This is like a gun going off in a cop’s holster while he’s texting and driving without a seatbelt. Yeah he’s a contractor but that doesn’t suddenly allow for such incompetence.

[sandeepkd]

I have worked with some of the experienced folks in federal space in the past, who were super smart, experienced and COSTLY from managements perspective. They had the ability to challenge the management on such things. Most of them have either retired, managed out or moved on. What you have here is not a reflection of the individual but the entire management chain. Its a race to make most money and at times these contractors are number of seats to fill at lowest possible cost.

[Forgeties79]

Totally agree

[MrDarcy]

The error and omission of not enforcing mandatory security training covering posting plaintext passwords to public sites for CISA contractors is itself an act of gross negligence.

So much so the contracting company’s insurer would cite it as the reason why the claim is not covered by their policy.

[morpheuskafka]

He worked for CISA. Surely there is either a security clearance with indoctrination and training, or at the very least, some sort of mandatory training/onboarding for all contractor staff?