Hacker News new | ask | show | jobs
RubyGems Fracture Incident Report (rubycentral.org)
94 points by schneems 76 days ago
4 comments

I can see a lot of time was put into the report, and it helps to have the detail, but in my mind it glosses over one of the most important parts: The dispute in the stewardship of the bundler and rubygems open-source projects.

As I understand it, Ruby Central controlled the rubygems and bundler github organizations, but did not "own" the projects in the traditional sense - the individual contributers have copyright on the code, and potentially even trademark rights. By then removing access of core maintainers to those projects, they removed access to something they don't "own" themselves.

This is all complicated by the fact that controlling a github organization or repo is different from owning the trademark or copyright. But some of the original maintainers clearly felt they had more of a right to those projects than Ruby Central did.

I believe not clarifying this before making these access changes was the biggest mistake that Ruby Central made, and it's not even mentioned in this report.

I don't have much skin in the game but as a passerby, I agree that the report obviously was made with a lot of time/effort but wouldn't dramatically change someone's view of Ruby Central or assure anyone this won't happen again. This is like writing an outage postmortem without really getting to the root cause and identifying what can be done to prevent in the future.
I think part of that is that it was written from the perspective of the bug that caused the outage ;)
There’s a ton of detail in the report so perhaps I missed it, but yes, the underlying structural/governance flaw of conflating a service, with the IP that runs that service, is a root cause here and seems insufficiently called out. The tragedy of misconception -> misconstruction -> misconfiguration is common when the bridge between governance and engineering is crossed.

The takeaway for the rest of is that separation of such concerns isn’t an abstract notion but needs to be reflected in the mechanical implementation of organisations, lest you get a train wreck later when perspectives don’t align and the whole picture crumbles.

> dispute in the stewardship of the bundler

This was never in dispute from the two parties. Ruby Central and "the maintainers" agreed from the beginning that it was collateral damage. The disagreement was what that meant and what to do with it. Hence the Sept 10 message from the Ruby Central Committee that they should move it to the Ruby core org (which IMO is long overdue).

The original plan (by the oss committee)was to move bundler to the Ruby org, that's what happened. When it did, the community generally like it (on HN and reddit comments).

> individual contributers have copyright on the code, and potentially even trademark

They're not the original authors of Rubygems so it's doubtful they have anything more than copyright on the code they contributed.

IIRC the original authors of rubygems are also the original founders of RubyCentral (chad fowler, david a. black, rich kilmer, jim weirich?), so probably the line was blurrier back then.
this is a good write up, I hope this really helps put the whole mess to rest.

    This incident involved many people over a rather long time scale, and it was important to detangle how people perceived events from how they actually unfolded. The subject matter is deeply subjective, and multiple failed attempts at writing this doc came as a result of aiming for objectivity, for blameless representation. Therefore, those named in this report are:

    - Full-time employees of Ruby Central
    - Part-time consultants who were involved in access discussions
    - Anyone who made an access change from September 10th-18th, 2025
    - Those who have already been publicly identified in the discourse

    Volunteer groups, including the Ruby Central Board and the Open Source Software (OSS) Committee, are listed, but their actions are represented as a group. Individual quotes from the OSS Committee are used without direct attribution when they represent a general consensus.


    Some execution failures and mistakes are individual, but the purpose of having a foundation and having an institution is that it can rise above individual limitations and provide robust, fault-tolerant systems. Therefore, these are our mistakes, collectively. And collectively we'll learn from them, but only if we face what happened, what we meant to do, and where we fell short.

    The hope is that by sharing this, we can provide some closure to the community and increase transparency

The undeniable effect of masking specific comments made by OSS committee members is to protect three members (2 current, 1 former) of Shopify's technical leadership around Ruby and Rails, who have all since left the committee. The one who left Shopify went to 37signals after.
> (2 current, 1 former) of Shopify's technical leadership

You'll have to take me on my word about it...but if I saw this as a driver of the issue I would have included it. I think saying "shopify was involved" is sort of like saying "people talked about RV at Rails World." Shopify is huge and hugely invested in Ruby's OSS ecosystem. I have my own critiques of the company, but not here. I think they're a net positive for Ruby OSS. I wish the general response was "more companies need to step up, I'll go talk to my leadership" rather than knocking these volunteers for their involvement. I've said elsewhere that if I were in the committee or in their shoes...I don't think the outcome would have been different (even if details would have). Also, you are welcome to disagree and have a different opinion.

I agree that it's best not to have situations like this. PSF bylaws "Section 5.15. Limits on Co-affiliation of Board Members." and similar rules are generally good at preventing the perception of conflict of interest (which is also important...that the perception alone can be damaging).

Right now, the committee is 100% one company (me). Because I'm the only one on it. Which is also a problem. Also, we're in a rebuilding/re-prioritizing phase with all of this...so it's hard to onboard while things are in flux.

You’d think that name, Shopify, would appear three times, once per employee/committee member. Or just once, to say the entire OSS committee was employed by Shopify, if we’re still identifying the group strictly as a group. Either would be fine.
> Or just once, to say the entire OSS committee was employed by Shopify,

Mike works at Basecamp (now and then). Based on comms I don't believe any of them acted on behalf of their employer i.e. no "team orders." Or if they did, they did so in ways that aligned with my perception of what I believed to be the correct read of the situation.

I also think that we (as humans) are much less incapable of knowing what things sway and influence our opinions than we think. We are much less capable of correcting for conflicts of interest than we would like. The study "tappers and listeners" is about adjusting for knowledge (curse of knowledge), but I think it applies to influence as well. Which is to say...I'm sure that everyone was influenced in many ways, but I felt they acted as individuals and reacted in real time.

There are other details of affiliations that I omitted from the former maintainers as well, that are true to state, and likely had some impact on their decisions ... but I used judgment to omit what I didn't think was fair or didn't think was immediately relevant. Not saying I got it all right all the time, but sort of chiming in to say "I'm not only omitting information in favor of one party." Yes, I'm biased...but I'm trying to correct for that bias. (A funny thing to state after just saying humans are bad at it, I know).

I read the article but I’m out of the loop enough to not understand Shopify and 37signals piece in this can you clarify? Is there another place I can be pointed to with that background information?
This is a disappointing look for Ruby Central. I have to get back to work, but their retroactive framing that Andre and Samuel's work on RV justified Ruby Central's subsequent actions is contradicted by their own admissions.

By their own admission, André is a contractor to Ruby Central. Contractors, especially under California law, have no contractual obligation of confidentiality to the other party unless there's a pre-existing agreement in place. They later admit in this "incident report" that they didn't have any legal agreements with André in place, so there's no basis for claiming André couldn't work on rv.

Samuel was an employee, not a contractor, but [California Bus. & Prof. Code § 16600](https://leginfo.legislature.ca.gov/faces/codes_displaySectio....) voids non-compete agreements—so even as an employee, he had every right to work on a competing project. There's no indication that he used Ruby Central's proprietary information to do so, and the report doesn't allege that. I have little doubt that if Samuel or André used proprietary information to develop rv, they would have already presented evidence of that.

Independent of the legalese, a "uv but for ruby" is a blindingly obvious thing to do, and Ruby Central doesn't get to lick the cookie and get upset when an independent contractor—Ruby Central's own characterization—does a thing they didn't fund.

My sourcing on this is that I run a 10-person business with employees in California. I'm not a lawyer, but I looked over enough of this paperwork that I feel confident opining on an internet forum.

That wasn't my read of what the postmortem is claiming. I didn't see a claim that anyone did anything illegal with proprietary information and the only legal question anyone raised was around a tangentially related proposal with user data[1]. I think the question about working on competing work is unfortunately more grey than most on HN would like, but even then nobody was fired/terminated for that. It sounds like people voluntarily left.

My biggest takeaway from this is the intermingling of opensource work/foundations/companies and employees/contractors/volunteers needs to be incredibly explicit. It sounds like everyone had very different expectations about what this group of people was (ranging from an exclusive club of influential ruby developers to a very formal, business-like foundation) and, as a result, each other's actions seemed hostile/strange/confusing.

[1] I actually think the comments about the proposal of selling the user data does a disservice to the postmortem. I think it invokes a much more emotional reaction from the reader than anything else and, while potentially interesting, seems like dirty laundry that doesn't change the lesson the postmortem teaches.

They are still trying to sue Andre, that is by definition claiming he did something illegal. The rest is just fluff to cover their insincerity (IMO).
The document didn't mention a lawsuit and I was just responding to the above comment with only the context of the postmortem and pointing out that this particular article didn't claim anything illegal happened. You and some others here might have much more context that I or other readers of this postmortem don't have.

I seem to remember there were some threats of legal action related to unauthorized access after this kerfuffle but I a) don't know what is going on with that, b) don't know what the law actually says about that and c) don't know if that is what you are referring to. If so, I think it is different than what the original comment alleged which was more about moonlighting/using proprietary information/competing. I think that topic is extremely complicated (e.g. I am not so sure moonlighting for a competitor while an employee is necessarily protected in California...) but that wasn't alleged in the postmortem anyway.

A couple of gentle corrections:

> The document didn't mention a lawsuit and I was just responding to the above comment with only the context of the postmortem and pointing out that this particular article didn't claim anything illegal happened.

You are correct that they did not make any claims, but the article did insinuate illegal behavior on the part of André and Samuel by selectively juxtaposing facts to imply wrongdoing without ever directly stating or saying that their behavior was illegal. For example:

1. André's first commit on RV is placed on the same bullet point as the Ruby Central-funded maintainer offsite, which implies Ruby Central's travel money subsidized a competing project's creation. 2. The `rubygems-github-backup` access token covering "all repos, including private repos" is introduced in the same timeline section as RV development, without any allegation it was used for RV. 3. The "Incident Lessons" section recommends adding an "Outside Business Activities" declaration policy, which only reads as a "lesson" if André's undisclosed side project is being framed as the problem in need of remediation. 4. The report states André "had intimate knowledge of the foundation roadmap" and "did not tell anyone in Ruby Central about this work until it launched". This frames nondisclosure of a lawful side project as a transgression. However, Ruby Central passed on this work, and even if they didn't, André has no obligation to tell Ruby Central about his work! 5. André's proposal to have his consultancy analyze RubyGems.org download logs is presented alongside an OSS Committee member raising PII and "reputational risk" concerns, casting a perfectly sensible rejected business proposal as something suspect.

By my count, Ruby Central makes roughly 10 insinuations throughout the report, but not once do they actually claim any of these constitute a transgression.

> I think that topic is extremely complicated (e.g. I am not so sure moonlighting for a competitor while an employee is necessarily protected in California...)

California is actually quite clear on this! Bus. & Prof. Code § 16600 voids non-compete agreements, and California courts have consistently read it broadly enough that working on a competing project during employment is protected. The line is whether you used your employer's proprietary information or resources to do it, not whether you competed. The report does not allege that Samuel or André used Ruby Central's proprietary information, and given how thoroughly they documented everything else, I'd expect them to have said so if they had evidence of it. Ruby Central is insinuating that working on RV in the first place is a problem, not that they crossed any legal or contractual line.

I might be reading it wrong, but it sounds like you and some others here are either more closely connected to the folks involved or at least have more context. I don't want to imply that I know better what the author or anyone at Ruby Central _actually_ believes or is doing, I'm just commenting on the article at face value. Whether the article is true, deceptive, or in between, I think there is still an interesting general lesson about organizations in it.

> You are correct that they did not make any claims, but the article did insinuate illegal behavior on the part of André and Samuel by selectively juxtaposing facts to imply wrongdoing without ever directly stating or saying that their behavior was illegal.

I think we just took away something very different from the article. I didn't read it that way, I read it more as "these two have already decided to move on to work on this without Ruby Central so it's pragmatic to cut off their access". We might just need to agree to disagree on what the article implies; perhaps we are just reading it with different boundary conditions.

Where we might agree is that repeatedly bringing up the selling user data proposal doesn't add anything to the story except to prejudice the reader against Andre. If it's to show that there was still some communication between Andre and others at Ruby Central, I would have kept it at that. Every time it got mentioned I winced.

> California is actually quite clear on this!

My understanding is quite different. There is a duty of loyalty an employee owes their employer and directly competing with your employer is clearly a breach. There is recent enough case law on this (at least covering terminating an employee for cause as a result). I don't have access to the materials from a previous employer that explained some of this but I did quickly find [1] which roughly agrees with my recollection (though I would not be willing to vouch for this particular site), namely "that Section 16600 has consistently been interpreted as invalidating any employment agreement that unreasonably interferes with an employee’s ability to compete with an employer _after_ his or her employment ends".

I'm not a lawyer (I assume you aren't either but at the very least you aren't _my_ lawyer) so I think it's not worth debating this further, we seem pretty firm in our beliefs on this one.

[1] https://www.aalrr.com/Business-Law-Journal/californias-polic...

EDIT: I want to acknowledge that one of the individuals here was a contractor and not an employee. I have no idea how that factors into moonlighting restrictions. I imagine it would be more limited and lean more on what that individual's exact role is at the company? I think my point still stands that my understanding is that the general situation for the average software engineer is more nuanced.

There is a lot of tension that the report seeks to either minimize or avoid. It’s also just really hard to express it in a report like this because there’s no real place for it if the goal is to look professional.

I think the RubyGems fiasco was a result of unresolved tensions. People chose not to be adults about and resolve the issues respectfully. IMHO, I think one of the main problems is that nobody was willing to spin up a core foundation to own critical infrastructure to the Ruby community which remains a problem.

I cannot find the blogposts I remember reading, but recall that there were some bad feelings about Ruby Together and Arko’s leadership of it before it was merged with Ruby Central. It appears these feelings never went away which is made very clear by the way that key Shopify engineers started posting after Ruby Central took over the RubyGems GitHub org [1].

Now combine this with dhh’s right-wing political posts and behavior, his extremely close relationship with the founder of Shopify (dhh is on the board of Shopify), a key Ruby Central donor pulling critical funding because he did not want his money going towards giving dhh more attention and you’re left with Ruby Central effectively being controlled by Shopify (which, as far as I can tell is still the situation) because that’s where all of its funding comes from now.

Frankly, the biggest thing this entire fiasco has shown me is that a lot of us are still a bunch of idiotic teenagers. Integrity and maturity is in short supply where it is needed the most.

[1] https://bsky.app/profile/rmfranca.bsky.social/post/3lz7alpob...

> I cannot find the blogposts I remember reading, but recall that there were some bad feelings about Ruby Together and Arko’s leadership of it before it was merged with Ruby Central.

This is one of the biggest ones that I remember:

https://web.archive.org/web/20180420220652/https://samphippe...

uv is Astral's onramp to paying customers. Without uv's tight integration with Astral's other tooling that they want to charge for, they wouldn't be able to sell anything. Building a business around doing the same for Ruby may be within their rights, but it's absolutely a conflict of interest working or contracted by Ruby Central. Removing them was an obvious move.
If this is a conflict of interest, then any Ruby core systems being controlled predominantly by members of the Shopify dev team is itself a conflict of interest. I am fine saying 'we need to make sure these libraries stay independent and community controlled', but that is so clearly not what was going on here. Believing that is just letting the RC FUD and PR control your thinking on the narrative.
I'm sorry but what are Shopify's business activities that directly compete with services provided/maintained by RubyCentral?

As far as arguments about community, Shopify IS the community by virtue of being the ones putting up pretty much all the money to keep this ship afloat.

If you don't have skin in the game your positions won't be taken seriously.

Depending on your point of view, Sidekiq either turned their back on the community or tried to start a coup by pulling funding just so they could morally grandstand.

That attitude is exactly the problem. Shopify does not 'keep the ship afloat' they are just a corporation using open source systems as the foundation of their business. Competition is not by definition the backing of a 'conflict of interest', it legally refers to a person or entity with a stake in a particular outcome having control of the means to achieve that which are not legally sound , ie compromise their judgment. I think Shopify's judgement of what 'is good for the Ruby community' is severely compromised by their corporate interests, and probably by their boards political interests as well. Hence, why they are trying so hard to justify removing Andre.
Do you have any idea how expensive it is to keep the infrastructure running? RubyCentral's operating expenses are in the millions every year and exceed their revenue.

Andre's removal is easily justifiable by his own (lengthy history of) sketchy behavior.

Since when is "open source" something businesses shouldn't be allowed to get value from or even have a stake in? These things are MIT licensed. That's free as in speech AND beer. If you don't like the freedoms of the license and how other people use them, don't use the license. If you don't like someone's stewardship, fork and maintain your own.

Shopify absolutely has an interest in preserving the primacy of the Rails-adjacent tooling with which rv would compete.