It was an old API access key that got leaked, not our account credentials. We're still investigating how and where the key got leaked, but bottom line, it should have been revoked ages ago.
2FA is great, but it doesn't cover API keys. Rotate your API keys!
MFA for all console accounts is the only right answer. If machines require credentials to do specific task or perform API calls then roles should be used.
> If machines require credentials to do specific task or perform API calls then roles should be used.
Even then, if the data must be considered highly valuable/immutable, then versioning/delete protection should be enabled for the S3 bucket(s) in questions. This requires the MFA token to be in the API call for the delete to succeed.
Any word on how those accounts are getting compromised?
Have they been complacent (easy password to guess, keys easy to be compromised (maybe in a public github repo)), or could there be some whole in the AWS secutiy model?
If there were a hole in the AWS security model for this, I think it'd be pretty obvious pretty quickly, given what happens when US-East takes a dive...
This happens constantly, and it's almost always through lack of best practices (as mentioned in higher up comment - IAM, MFA, etc.).
This sucks... I am happy we just put our search cluster on elasticbeanstalk atm, but I wish we had more services like this running.. good news is new security practices will hop up everywhere because of this.
Good that it's pretty easy to change elastic search provider with little downtime. I'd recommend checking out found.no. We've been pleased with performance and stability. Heck - I can't recall any downtime at all.
qbox.io is another hosted Elasticsearch solution. I have never used them, but one of their developers is very knowledgeable and active on the mailing list.
our site http://www.violetgrey.com went down because of this. Luckily we were able to reindex pretty fast before their backups kicked in. Any ideas on how to have fallbacks in such cases?
AWS Multi-Factor Authentication (MFA):
http://aws.amazon.com/iam/details/mfa/
AWS Identity Access and Management (IAM):
http://docs.aws.amazon.com/IAM/latest/UserGuide/IAMBestPract...
Managing your AWS API Keys:
http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSG...
Go a step further with your AWS API keys and use AWS' API access logging (CloudTrail):
http://aws.amazon.com/cloudtrail/
Don't get burned. Check your stuff out.