[matthewisabel]

Hey from the GitHub team. Outages like this are incredibly painful and we'll share a post-mortem once our investigation is complete.

It stings to have this happen as we're putting a lot of effort specifically into the core product, growing teams like Actions and increasing performance-focused initiatives on key areas like pull requests where we're already making solid progress[1]. Would love if you would reach out to me in DM around the perf issues you mentioned with diffs.

There's a lot of architecture, scaling, and performance work that we're prioritizing as we work to meet the growing code demand.

We're still investigating today's outage and we'll share a write up on our status page, and in our February Availability Report, with details on root cause and steps we're taking to mitigate moving forward.

[1] https://x.com/matthewisabel/status/2019811220598280410

[Etheryte]

Literally everyone who has used Github to look at a pull request in say the last year has experienced the ridiculous performance issues. It's a constant laughing point on HN at this point. There is no way you don't know this. Inviting to take this to a private channel, along with the rest of your comment really, is simply standard corporate PR.

[matthewisabel]

Yes agreed it's been a huge problem, and we shipped changes last week to address some of the gnarly p99 interactions. It doesn't fix everything and large PRs have a lot of room to be faster. It's still good to know where some worst performance issues are to see if there's anything particularly problematic or if a future change will help.

[ibrahima]

FWIW, I find the new React-based diff viewer worse than the old server-rendered page. I disabled the preview for this reason. It does have some nice features but overall it feels more finicky. I would think that in theory this should be better at handling large diffs but I'm not sure that that's the case, and at least the UX feels more choppy.

[Etheryte]

That's financialization at play. When you render and syntax highlight the diff on the server, Github pays the cost, if you do it on the client side, the cost is paid by the client. At Github's scale it's probably a large enough of a difference that they decided the reduced customer experience is worth it.

[jrochkind1]

I look at pull requests daily, I haven't encountered the problems you speak of, not sure what they are.

[rnhmjoj]

I have been using GitHub since 2011 and it's undeniable that the performance of the website have been getting worse. The new features that are constantly being added are certainly a factor, but I think the switch to client-side rendering that obviously shifted the load from their server to our browsers and also tend to produce ridiculously large and inefficient DOMs[1] is the main cause.

If you want a practical example, here you go. I'm a Nixpkgs commiter and every time I make a pull request that backports some change to the stable branch, GitHub unprompted starts comparing my PR against master. If I'm not fast enough to switch the target branch within a couple of seconds it literally freezes the browser tab and I may have to force quit it. Yes, the diff is large, but this is not acceptable, and more importantly, it didn't happen a few years ago.

[1]: https://github.com/orgs/community/discussions/111001

[whstl]

It's insulting to see the word "progress" being used when the PR experience is orders of magnitude slower than it was years ago, when everyone had way worse computers. I have a maxed M5 MacBook and sometimes I can barely review some PRs.

[materielle]

Hopefully the published postmortem will announce that all features will be frozen for the foreseeable future and every last employee will be focused on reliability and uptime?

I don’t think GitHub cares about reliability if it does anything less than that.

I know people have other problems with Google, but they do actually have incredibly high uptime. This policy was frequently applied to entire orgs or divisions of the company if they had one outage too many.

[danudey]

For what it's worth, I doubt that people think it's the engineering teams that are the problem; it feels as though leadership just doesn't give a crap about it, because, after all, if you have a captive audience you can do whatever you want.

(See also: Windows, Internet Explorer, ActiveX, etc. for how that turned out)

It's great that you're working on improving the product, but the (maybe cynical) view that I've heard more than anything is that when faced with the choice of improving the core product that everyone wants and needs or adding functionality to the core product that no one wants or needs and which is actively making the product worse (e.g. PR slop), management is too focused on the latter.

What GitHub needs is a leader who is willing and able to say no to the forces enshittifying the product with crap like Copilot, but GitHub has become a subsidiary of Copilot instead and that doesn't bode well.

[tayo42]

> people think it's the engineering teams that are the problem;

It could be, some people are just terrible at their job. Lots of teams have low quality standards for their work.

Maybe that still comes down to leaders but for different reasons. You can ship useless features without downtime.

[tjwebbnorfolk]

Permitting terrible engineers to continue to work for you is a management problem.

[tayo42]

Sort of I think. There's a culture aspect to it too. Everything is blameless, there's no reason to not mess up.

[rasz]

>I doubt that people think it's the engineering teams that are the problem

Did you forget Microsoft engineering response to Casey Muratori "Extremely slow performance when processing virtual terminal sequences"?

"I believe what you’re doing is describing something that might be considered an entire doctoral research project in performant terminal emulation as “extremely simple” somewhat combatively."

https://github.com/microsoft/terminal/issues/10362#issuecomm...

followed by Casey producing evidence for his 'extremely simple' claim in couple of days.

[cebert]

Can you guys stop adding new features for a while please and just make what’s there more reliable?

[DANmode]

GitHub will prioritize migrating to Azure over feature development

4 months ago on HN: https://news.ycombinator.com/item?id=45517173

[dickersnoodle]

Every developer has had this fight with management and most lose.

[tosti]

Why does it trigger a usage limit whenever I search anything at all? Sometimes I can get through but when I go back one page, boom error.

[Akronymus]

Oh I assumed that was just me/because of a linux OS.