[diggan]

Why is the title of the post-mortem "GitHub Outage"? It makes it sound like Lovable somehow brought down GitHub, when in reality it seems like they were rate-limited by GitHub for creating lots of repositories, then got their GitHub App completely blocked for breaching the Terms of Service.

> Incident report for the GitHub outage on January 2-3, 2025

Writing it like that looks like you're pushing the blame on your downtime/outage to GitHub, like they're responsible for your application to be up, instead of taking full responsibility for it.

[mschuster91]

> Writing it like that looks like you're pushing the blame on your downtime/outage to GitHub, like they're responsible for your application be up, instead of taking full responsibility for it.

Well, they did what they were supposed to - they explicitly asked Github what they were up to, Github gave an explicit "we're ok with this, go ahead", and once Github sees that, whoops, it's causing errors they don't even bother to check if there are support tickets open with the customer, they just go and disable their access.

[diggan]

But that's true for any 3rd party you'd depend on. Everything will work until it doesn't. Doesn't mean you're less responsible for your project being down.

A title like "GitHub caused our outage" would still make it clear the downtime wasn't the direct action of anyone on the team, yet still take responsibility over that it happened. Instead, labeling it "Incident report for the GitHub outage" just seems like straight up blaming someone else.

[Mathnerd314]

Well, Github explicitly took responsibility. The first action Github did once Lovable reached out for support was "reinstate our app and apologize for the issues it caused us and our users."

And no, you are not responsible for every 3rd party service you use. Some services are unavoidable, some services are just nice-to-have, but if you can't trust a service to perform its advertised function, it is the service's fault.

[diggan]

> Well, Github explicitly took responsibility. The first action Github did once Lovable reached out for support was "reinstate our app and apologize for the issues it caused us and our users."

No, GitHub won't ever take responsibility for what you have between you and your projects/apps/companies users. Nor did they do so in this case.

If I use service X for doing Y, and I write an email asking if it's OK that I upload 1000s of files every day, even if they say OK today, but then next week turn around and say "Nah", I'm still responsible for my users, who trust me and my team. Service X has no responsibility towards those users at all. It sucks though, I agree with that.

> And no, you are not responsible for every 3rd party service you use. Some services are unavoidable, some services are just nice-to-have, but if you can't trust a service to perform its advertised function, it is the service's fault.

Besides DNS and BGP I suppose, what services are "unavoidable" exactly? Git hosting isn't some arcane distributed network technology needing years of reading/experience to understand, the CLI ships with a web ui you can basically copy-paste to have your own Git hosting.

I'd say you are responsible for everything that you use and depend on. And if you think "Ah, I'll just chunk 10K repositories at GitHub a day, they say it's fine" and then don't have any plan in case GitHub suddenly says it isn't OK, you are responsible for the falloff if shit hits the fan.

[Mathnerd314]

Well, the app store, for example. Sure, it is of course a good idea to comply with the app store policies to the extent possible, but ultimately there not much you can do to prevent Google or Apple saying "we don't like this app" and pulling it. For example with the UTM emulator. So how then can Google or Apple making such a decision be "your responsibility"?

As another example, let's say you build a house in a hurricane-prone area. It's your responsibility to ensure the owner buys hurricane insurance, as mandated by law. It's not your responsibility to build a nuclear-bunker-grade house that is impervious to hurricanes. It is easy to point the finger and say "you should have thought of that", but in practice it is easier to deal with such catastrophes as they happen.

[AstralStorm]

Besides the fact that you also need packet traversal via a multitude of layer 3 protocols a lot of which would pass through some corp cloud. And the net neutrality is dead in the USA.

As for the repositories, there are no good SLA terms for such storage. Nobody offers this service because it's expensive to offer. Not on this scale. So if you need it, you have to invest a lot in hardware or colocation or specialized clouds. Plus admin. A whole datacenter, and suddenly you're an ant vs three goliaths.

[hiatus]

> but if you can't trust a service to perform its advertised function, it is the service's fault.

Your customers don't care if it's the service's fault and will blame you.

[cudgy]

If I read it correctly, it was a support person that provided them with assurance. Not an executive or vice president or manager or vp of sales. GitHub did not give them permission nor their approval; it was a single person in support department.

Who relies on support people to determine the basis of their business when it’s obvious that they were concerned with the high usage rate and that it might cause problems for their customers?

[kgeist]

Oh yeah, support staff aren't always aware of everything. For example, with one of our latest features, we didn’t have time to add a UI option to disable the feature. The expectation was that support staff could disable it via their special admin panel upon user request. However, I accidentally discovered that when users asked to hide the feature, tech support told them it wasn’t possible! It turned out the tech support lead forgot to share that information with the team.

As for the OP, they should have conducted load testing and implemented rate limits on their end, rather than blindly relying on someone’s word that GitHub was ready to handle all their product's load for free.

[arccy]

sounds like a pretty sane thing to do: github protects the majority of their customers from instability caused by a few.

unless you're a super important partner, the people on call might never have heard of your little app and just decide that it's the only safe thing to do to protect the reliability of the system.