[tonywebster]

Don't be so quick to dismiss this one. This is notable for several reasons, including (1) it wasn't his mistake, but rather that of a Visual Studio bug not following his instructions to make a private repo on GitHub; (2) the speed in which this happened (minutes), and (3) it has useful analysis into some sorely lacking functionality in AWS that lets this continue to happen.

[nathancahill]

I don't know. Even for private repos, it's bad practice to commit private keys to source control.

[zeveb]

> Even for private repos, it's bad practice to commit private keys to source control.

Is it, though? Committing them to the same repository as one's code lives in, sure, but committing them to a separate production-deploy repo seems okay to me (although I'd much prefer that private repo never to hit a centralised service like GitHub).

[curun1r]

> Is it, though?

Yes, it is. Period. Full Stop. Don't ever check access keys into any repo, public, private or even self-hosted. AWS needs to do a better job of making you realize that access keys are like 100-year-old sticks of dynamite and should be handled with an equal amount of care considering they can cause a similar amount of damage.

To their credit, they basically noted this when they changed the way access keys are handled in the command line tool[1]. Quoting:

  An important point is that the default location for the credentials file is a user
  directory. It's no longer part of a project file structure, such as an app.config file
  (.NET) or .properties file (Java). This can enhance security by allowing you to keep the
  credentials in a location that's accessible only to you, and it makes it less likely that
  you'll inadvertently upload credentials if you upload a project to a developer sharing
  site like GitHub.

[1] http://blogs.aws.amazon.com/security/post/Tx3D6U6WSFGOK2H/A-...

[balls187]

> Yes, it is. Period. Full Stop. Don't ever check access keys into any repo, public, private or even self-hosted.

Generalizing here (and for the record I agree with your point), is that to a developer, source code is as valuable as anything else, and if a private repo is secure enough for source code, it should be secure enough for access credentials.

I speculate that this is a by product to moving towards a developer-centric engineering culture.

[zeveb]

> Don't ever check access keys into any repo, public, private or even self-hosted.

So, how do you propose to handle version control of your exact production configuration? Do you think that recording one's production configuration is a valuable (or even useful) technique? Do you propose to have a single file with production configuration relying only on point-in-time backups for history? That gets us back to the bad old days…

A local or self-hosted repo is 'a local location that's accessible only to youation that's accessible only to you.'

[curun1r]

> So, how do you propose to handle version control of your exact production configuration?

Version control is perfectly acceptable for that, but the key point is that access keys aren’t part of your production configuration. I personally don’t allow my team to have long-lived AWS access keys that have access to production. Production is only changed via an assumed role requiring MFA or an IAM instance role.

If this kind of setup is too complex to create on your own, solutions like Vault (https://vaultproject.io) can make it much easier to get right.

[zeveb]

> Version control is perfectly acceptable for that, but the key point is that access keys aren’t part of your production configuration.

Why not? If your production configuration is stored as static-file in version control, then either those credentials have to live in that version control, or they have to live outside it, as a separately-backed up file of configuration variables, which are then fed to a static-file-generating tool, which means a) that one has to select a templating system and maintain those separate templates and b) that there's another step involved in developing the prototypes for those templates. This isn't a bad course of action, of course, but it does have its costs (just like storing production configuration—including credentials—in version control has its costs as well as its benefits).

> I personally don’t allow my team to have long-lived AWS access keys that have access to production.

Of course! Developers shouldn't have access to production systems, period. But operators can, and must, and I've no problem with them having access to their own version-controlled repository of information that no other team can access.

> Production is only changed via an assumed role requiring MFA or an IAM instance role.

Of course! I'm certainly not arguing for personal credentials to be stored in an operations repo.

[nathancahill]

I use Ansible Vault if it's imperative to store keys in a repo, and I've been really happy with it. StackExchange has something similar called Blackbox[0]. I remember reading about a Ruby gem too, but I can't find it now.

[0] https://github.com/StackExchange/blackbox

[zeveb]

> https://github.com/StackExchange/blackbox

That looks pretty cool, at least at first. I'll have to take a deeper dive into it.

I like that it looks to be all shell scripts. Wish it were sh rather than bash, but c'est la vie.

[cptnbob]

AWS SDK for .Net supports named profiles for VS so your source check in only contains a profile name. If you hard coded your IAM or account keys or stuck them in the app.config you're simply doing it wrong. There is no excuse. It's all here: http://docs.aws.amazon.com/AWSSdkDocsNET/latest/V3/Developer...

Also if you use a proper IAM profile locked down to specific resources then you wouldn't expose your entire account. The author stated he didn't use EC2 so why wasn't the key/secret pair an IAM account with a policy set for minimal access?

What I'd worry more about is that the VS bug exposed private source code, data and proprietary intellectual property.

We use github and I worry every day someone will public fork one of our repos by accident. That would be a grave fuck up but it's waiting to happen. We should have stuck with centrally controlled active-directory integrated SVN from a security perspective (even if it is a pain in the ass).

[gregmac]

Well, it is partially his mistake, which he acknowledges:

> I am certainly not innocent here and some mistakes were made on my part. [...] To this end, having encrypted access keys or excluding configuration settings from GitHub would have prevented the AWS charges - and this is certainly the approach I would take from now on.

[morder]

In this case it wasn't Visual studio but an extension created by Github that caused the problem.