[ctoth]

AWS account root access on a language package registry for 11 days. Not EC2 root - AWS account root. Complete control over IAM, S3, CloudTrail, every-damn-thing.

They're claiming "no evidence of compromise" based on CloudTrail logs that AWS root could have deleted or modified. They even admit they "Enabled AWS CloudTrail" after regaining control - meaning CloudTrail wasn't running during the compromise window.

You cannot verify supply chain integrity from logs on a system where root was compromised, and you definitely can't verify it when the logs didn't exist (they enabled them during remediation?).

So basically, somebody correct me here if I'm wrong but ... Every gem published Sept 19-30 is suspect. Production Ruby applications running code from that window have no way to verify it wasn't backdoored. The correct response is to freeze publishing, rebuild from scratch (including re-publishing any packages published at the time? Ugh I don't even know how to do this! ) , and verify against offline backups. Instead they rotated passwords and called it done.

[tptacek]

Isn't the subtext of this post pretty clearly that the unauthorized actor was Andre Arko, who had until days prior all the same access to RubyGems.org already?

The impression I have reading this is that they're going out of their way to make it clear they believe it was him, but aren't naming him because doing so would be accusing him of a criminal act.

[ctoth]

Let's say that they are 100% correct, we parse the subtext as text, it was totally him.

We still do not know the critical details of how (and when) he stored the root password he copied out of their password manager (encrypted in his own password manager? on his pwned laptop? in dropbox? we'll never know!) therefore the whole chain of custody is still broken.

[tptacek]

The leading contender to replace RubyGems has Andre Arko as a charter member, so this all seems very salient.

[busterarm]

Right but that speaks more to Andre's character, IMO.

Why are you copying a password out of a shared vault that should only be used in break-glass type scenarios? It's that's not planning for possible malicious action in the future, I don't know what is.

You can try and excuse it as having your own break-glass for the break-glass, but that's on the spectrum between irresponsible and incompetent.

Again, if the accusation is true, removing him was justifiable from any possible perspective you might have.

[phoronixrly]

The other subtext is that they literally have no idea how to run rubygems securely... And what to do in case of a security incident...

[tptacek]

I'm addressing the question of whether we all had better assume all the RubyGems published after this incident were compromised, and my response is "that is probably not rational since the actor in this scenario had all this access legitimately just days beforehand". The rest, I don't care.

[phoronixrly]

Look, it's enough to know that Rubygems did not require 2FA before August 2022. There were gems with millions of downloads with owners without 2FA on their accounts. I think your initial assumption is pretty safe even without the ongoing fiasco.

[JoshTriplett]

The other other subtext is that this sure is an effective distraction from their governance problems, and muddies the waters. Given the utter lack of trust I have for anything the Ruby Central folks say at this point, given the amount of spin and misinformation they've spread already, my default assumption is that this is an excuse to malign someone who may well have had legitimate access, in the process of claiming that you're locking things down, which was always the excuse being made for kicking people out.

[JoshTriplett]

Update: https://andre.arko.net/2025/10/09/the-rubygems-security-inci... is pretty much exactly the kind of thing I expected here. Person with legitimate access doing their job, organization flailing around in the process of kicking people out that should never have been kicked out in the first place.

[tptacek]

He changed the AWS root account password; RC implies they had to go through a reset flow to recover the account. This apparently went on for more than a week. I don't know how to reconcile what Arko is claiming with what RC is claiming.

[sylens]

Arko believed he was in the right to do so, and while he probably should've reached out sooner to notify them of the "precaution" he was taking, the fact that they didn't notice for almost two weeks shows how unserious they are about security

[hitekker]

At this point, it looks like everyone involved, not just RubyCentral, contributed to the governance problems over many years https://archive.md/SEzoV

> Regarding Arko’s blog post about his removal, McQuaid [Homebrew Maintainer] told me it’s good that Arko is crediting other people for their contribution and that he’s following open source principles of community and transparency, but that “his ‘transparency’ here has been selective to things that benefit him/his narrative, he seems unwilling or unable to admit that he failed as a leader in being unwilling or unable to introduce a formal governance process long before this all went down or appoint a meaningful successor and step down amicably.”

[landdate]

It seems to me like the inherent trust in open source software is a big problem. Reliance on software maintained by strangers, sometimes just one individual, and not reading/understanding the code before running it.

[placardloop]

CloudTrail logs for the last 90 days are enabled by default, cannot be turned off, and are immutable, even by root. If you view this “event” as starting when Arko was supposed to have their access terminated, that’s within the 90 day window and you can indeed trust the logs from that period.

[ctoth]

CloudTrail's 90-day immutable Event History only logs management events (IAM changes, instance launches, bucket creation). It does NOT log:

* S3 object reads/writes (GetObject, PutObject) - these are "data events" requiring explicit configuration[0]

* SSH/RDP to EC2 instances - CloudTrail only captures AWS API calls, not OS-level activity[1]

With root access for 11 days, someone could modify gem files in S3, backdoor packages, SSH into build servers - none of it would appear in the logs they reviewed. Correct?

[0] https://docs.aws.amazon.com/awscloudtrail/latest/userguide/l...

[1] https://repost.aws/questions/QUVsPRWwclS0KbWOYXvSla3w/cloud-...

[placardloop]

SSH is totally irrelevant here. Having AWS root account access doesn’t give you any ability to SSH to or otherwise access running instances. You could access data on those instances by cloning the EBS volumes or modifying build pipelines or changing network access or similar, but these would all show up in CloudTrail even without data events enabled.

For S3 objects, you don’t necessarily need data events to identify if tampering happened. S3 objects are immutable as well, so if any changed you would see that reflected in the creation date and new hashes that S3 attaches as tags, which you can correlate with application logs to see if they match up or not. It’s not as simple as data logging, sure.

But you’re also missing the key component here that they did not say they only just enabled CloudTrail logs, they’re saying they just now enabled CloudTrail log alerting. We don’t have any idea if data events were enabled or not, or if things like flow logs were enabled or not, or what other investigation tools they have running at the application layer. However, even if none of existed, there’s still a lot more audit-ability of events that happen in an AWS account than you’re implying, even the root account.

[knert]

I’m not sure this is true. The EC2 web console terminal drops me right into root on any of my instances.

[placardloop]

Ahh you’re right, there are some that just initiate a connection via something like Session Manager, but those connections where AWS initiates the connection for you are logged in CloudTrail, even without data events, and root doesn’t give you any ability to directly SSH into an instance outside of those methods (you cannot, for example, use root to find out what the private keys are for logging into an instance) so we’re back to the fact that any such access would be auditable.

[dragonwriter]

> Having AWS root account access doesn’t give you any ability to SSH to or otherwise access running instances.

It does if access credentials to those instances are stored in any AWS service in the account (such as in Secrets Manager).

[sylens]

Even without data events, there are clues you could infer from management events that somebody is going through buckets. ListBuckets and GetBucketLocation are management events and are seen often by someone enumerating buckets to exfil data from. The fact that this actor was someone very familiar with the account might make this a moot point though as they would know exactly which bucket to go to and what region its in.

[ctoth]

Thinking about this a bit more... it sure is interesting that around the time of a competing project launch that something just happens which might reasonably completely compromise trust in the previous incumbent, isn't it? Odd!

[dmix]

That was intentional according the Joel Drapper who leaked this incident, he wanted to make Ruby Central look bad

https://www.reddit.com/r/ruby/comments/1o2bxol/comment/ninly...

>> Why did Joel give so little time of advance notice before publishing his post revealing Andre’s production access? That struck me as irresponsible disclosure, but I may have missed something.

> I decided to publish when I did because I knew that Ruby Central had been informed and I wanted the world to be informed about how sloppy Ruby Central were with security, despite their security posturing as an excuse to take over open source projects.

> What I revealed changed nothing about Ruby Central’s security, since André had access whether I revealed that he did or not. When you have security information that impacts lots of people, you publish it so they can take precautions. That is responsible disclosure.

[picadi]

how can you trust gem.coop isn't already mining request logs + IPs to try and monetize lists of companies using specific packages & versions — besides the privacy/ethical concerns it is super useful data for hackers looking for vulnerable apps

no single person should have Github owner + AWS root password for a major language's package manager and ecosystem just sitting around on their laptop while they fly around to different conferences (as Andre seems to have done while showing off he still had the login to rubygem's AWS root account while in Japan)

[coredog64]

Not going to bother reading the article, but will chime in here that the recommendation from AWS is to have a separate security account within your organization that only holds your CloudTrail logs. This does potentially double your cost, as you only get one CloudTrail for free, and it's very useful to have an in-account trail for debugging purposes.

Organizations are also useful because you can attach SCPs to your accounts that deny broad classes of activities even to the root user.

[shevy-java]

You are actually right, I didn't think of this before.

How can they ensure that nobody else did any tampering?

It seems RubyCentral did not think this through completely.

[blibble]

> It seems RubyCentral did not think this through completely.

this is the problem when you fire all the maintainers who do anything

[busterarm]

You can set up EC2 instances in a way that that just having AWS root access doesn't give you ssh/console access to the instances. You can still do things like Run Command but that leaves a very obvious trail (although even this is preventable with enough effort).

Also you can enable cloudtrail log validation which can ensure you know if you're looking at tampered logs or not.

Really it all depends on how their accounts are set up. Unless you know the operational details you can't make a call here.

I've run a multi-million dollar/year AWS Org for the last decade or so and setting things up this way is kind of brass tacks.

[ctoth]

So you almost certainly know that a lot of IaC tooling has terrible defaults like enable_log_file_validation being set to false. Based on the quality of their credential management and what else you can read from this blog post, how much would you wanna wager they did it right?

[busterarm]

I assume everyone is professional until they prove otherwise.

Based on how things have been described on both sides, it actually sounds like they do a pretty good job. Oversights happen -- we're all human -- and this access was already limited to a small single-digit number of people. Given the history, it's reasonable that Arko would have had this high of a level of access and the oversight was in forgetting that when removing him.

Also it's reasonable to assume that people with that access wouldn't do something criminal/malicious, and if they did, while annoying, the situation is very easily recoverable. Especially if you're using IaC tooling as you mentioned.

If you're already taking the position that Ruby Central are "the bad guys" it's easy to assume that they're doing everything wrong, but that would be a mistake.

[frenchtoast8]

I believe this is a scenario where AWS recommends multiple accounts.

1. Create another "management" AWS account, and make your other AWS account a child to that.

2. Ensure no one ever logs in to the "management" account, as there shouldn't be any business purpose in doing so. For example, you should require a hardware key to log in.

3. Configure the "management" account to force children account to enable AWS Config, AWS CloudTrail, etc. Also force them to duplicate logs to the "management" account.

Step 2 is important. At the end of the day, an organization can always find a way to render their security measures useless.

[0cf8612b2e1e]

2) Surely, someone needs access to the account. How do you prevent those with access from using it? Security feels like turtles all the way down where you ultimately have to trust a few people to do the right thing.

[zrail]

The only reason someone would need access to the management account would be maintaining child accounts and IAM roles or reviewing logs, none of which should need root.

[yargzeblog]

IMO the only way to avoid doing a total rebuild is to have Andre Arko:

1. Admit that he was the unauthorized actor (which means he's probably admitting to a crime?) 2. Have him attest he didn't exfil or modify the integrity of service while committing a crime.

If I was Ruby Central I would give clemency on #1 in exchange for #2 and I think #2 helps Andre Arko.

[ctoth]

So you would expect people to accept that the entire root chain of custody for the Ruby supply chain is attested by ... A guy saying he didn't do anything bad? I have a cool cryptocurrency you might wanna check out that I definitely don't have a backdoor to!

[LamaOfRuin]

If Andre doing that was criminal, it seems quite possible that their original takeover of the github organization was also criminal?

I have been waiting to hear if there would be any civil action on it since it's not at all clear they had any rights to do most of what they did.

[tptacek]

No, that's not at all clear. Ruby Central owns the AWS account for which Arko is (pretty clearly) being accused of changing the AWS root account password after having his access revoked.

I don't think for a second Arko will be charged, but there isn't a "nuh-uh, you did this gross thing in our open source community" defense for 18 USC 1030.

[LamaOfRuin]

I didn't say it was clear, and I never said there was a defense. I implied that the wronged party in one case might want to be careful about raising the specter of liability or criminality.

[SAI_Peregrinus]

Ruby Central isn't capable of giving clemency. They could refuse to testify in any prosecution, but they don't get to pick whether a relevant attorney general or district attorney decides to prosecute.

[dragonwriter]

> They could refuse to testify in any prosecution

Legally, they can state a preference not to testify, but they couldn't (legally) refuse to if issued a subpoena. (And, even more emphatically, they couldn't accept a good or service from the person who might be charged in exchange for not reporting the crime, or for refusing testimony.)

[dragonwriter]

In the US, at least, private parties can not grant immunity from prosecution for a crime (only public prosecutors of the jurisdiction against whose laws the crime was committed can do that), and they may face legal jeopardy in agreeing, or even offering, not to report a crime in exchange for some good or service of value, as that is the definition of blackmail.

[florkbork]

I hate to ask; but

- Account created 14 hours ago.

- Posts article crammed full of accusations

- Has a strong well formed opinion about "it's a crime", but didn't? read the content where the subject of the accusations has.... Already disclosed they had access in both private and public.

My account is also very new, because I have opted to discard my previous ones. I have used it to comment predominantly on this topic, as I sympathise with the maintainers.

So in the interests of making a similar disclosure is there any chance you are affiliated with RubyCentral through a business relationship with them, their legal counsel, a marketing or PR agency or anything of that nature?

[akerl_]

Given the context of the post, it seems like "Enabled AWS CloudTrail, GuardDuty, and DataDog alerting" means "enabled alerts via CloudTrail, GuardDuty, and Datadog", not "enabled Cloudtrail logging". Otherwise the comment about reviewing Cloudtrail wouldn't make sense.

[ctoth]

So the attacker turns logging off (was log file validation enabled? usually isn't in Terraform ) which does not fire an alert because there is no alerting. Then does their bad stuff ... Then modifies the logs (which are in an S3 bucket on the compromised account, remember!) Then they turn logging on? The whole point is alerts go outside AWS. They go to like, your inbox or pagerduty or whatever. If they had no alerts then what use are their logs, which could have been modified? Do you think they set up cross-account logging or had enable_log_file_validation set to true?

[arianvanp]

You can't enable or disable AWS Cloud Trail as far I know?

You can enable the persistent storage of trails. But you can always access 90 days of events regardless of that being enabled

[ctoth]

Only management events, see https://news.ycombinator.com/item?id=45532772

[frenchtoast8]

This was my understanding as well, but earlier I couldn't find any documentation to prove this so I never wrote a comment.

CloudTrail can be configured to save logs to S3 or CloudWatch Logs, but I think that even if you were to disable, delete, or tamper with these logs, you can still search and download unaltered logs directly from AWS using the CloudTrail Events page.