[edgyquant]

I’m not sure at what organization that is true. My company lives out of GitHub and Jira and I’ve hardly noticed the three month surge. GitHub would have to do a lot worse to get many companies to want to host their own services. This is the argument people have said about the cloud from day one.

People want to know it isn’t their problem, that makes cloud computing (and things like GitHub) worth their weight in gold. I have real problems to solve I don’t want to deal with a git repo manager on top of that.

[thaeli]

Also, looking at this it seems like GitHub isn't doing the common SaaS thing of just lying on their status page. Many providers, both internal and external, would look a lot worse if they had honest status pages.

[mirekrusin]

They are green for good 15 minutes from first moment i see problems, not the first time, it happens actually quite often. Maybe that's the time they need to confirm/cross check/write status update, don't know.

[georgemcbay]

While quicker reporting would be better, 15 minutes is anecdotally a lot better than I see from most other services where their status pages will report all-clear hours into full outages.

[thaeli]

Yeah, I'm legit impressed with a 15 minute time here.

[judge2020]

They probably allow regular SREs to trigger an incident on the status page on their own, when the likes of AWS and other bigger cloud providers are rumored to need approval from a VP[0] to update the status page.

0: https://news.ycombinator.com/item?id=29475756

[jhugo]

Several of the recent outages were much longer (at least for us, here in Asia) than they admitted on their status page. In one case I started work, noticed I couldn't push to or pull from GitHub, that situation persisted all day, and around 5pm local time (so morning-ish in the US) suddenly their status page acknowledged the problem and a discussion started on HN.

[ishanjain28]

They do intentionally or not lie about this on their status page. From December 25th to December 31st 2021, Github actions had network problems almost every single day for hours and the status page was green out through out that period.

Same thing also happened few months back.

It feels like they do this manually and it's only done when enough people are effected.

[wnevets]

> I’ve hardly noticed the three month surge.

This has been my experience as well. I don't know if that means GitHub is being overly transparent about issues or I've just been lucky but I would hate if people punished services for being transparent and informative on their status pages.

[zenexer]

GitHub's outages have hit me hard over the past week or so. I don't think it's a matter of them being transparent--if anything, I was hitting errors well before their status page updated. Yesterday it was completely unusable for much of my workday, and today tasks that normally take me a few minutes have been taking hours.

[SkyPuncher]

> I’m not sure at what organization that is true. My company lives out of GitHub and Jira and I’ve hardly noticed the three month surge

These have been minor inconveniences for us - at worst. Most of the time it simply means people jump to something else then come back later in the day.

Failing tests and PR feedback cycles are more of a blocker to our team than these outages.

[gjulianm]

At my organization it's always been true. Setting up GitLab is fairly easy, in my company we do it and it's cheap (on-prem hosting is basically zero, and we had the IPs/domains already) and it hasn't given us too many headaches. I think last time I had to do something was maybe a few months ago when I restarted it so that it picked up the updated SSL certificate.

[nightpool]

Self-hosted GitLab got a good callout yesterday from Microsoft, it appears to be a favorite of LAPSUS$: https://www.microsoft.com/security/blog/2022/03/22/dev-0537-...

Self-hosting always increases the operational burden of making sure your systems are secure. Maybe you have the engineering resources to spend on patching everything immediately and conducting in-house pen tests, but for most companies it's much, much more secure to let the software's developers host it as well.

[temp8964]

Not necessarily. Self-hosted services are protected by company firewall / VPN. They can setup very restrictive network access. They don't have the same level of risks as public services like GitHub or GitLab.

[hhh]

Establishing an entry point via VPN is Lapsus$ primary first step.

[Melatonic]

Except that the software developers hosting is also a much, much bigger target and you generally do not have any real control over how often they are patching either.

[skeeter2020]

>> Setting up GitLab is fairly easy, in my company we do it and it's cheap (on-prem hosting is basically zero, and we had the IPs/domains already)

In what tech company is hosting or domains the main cost centre? Many companies spend more on a single hour of a dev's time than their entire GH monthly bill.

[capitol_]

I think we pay about $10 per developer per month for github, and with about 1000 developers I would love that hourly rate.

[kleebeesh]

...What? $10 x 1000 = $10k / month. $10k x 12 = $120k. That is a new grad software engineer salary in any US city. You'd pay more than that for a single dev with the devops and security experience to keep GHE running and patched for 1000 devs.

[cortesoft]

The person was replying to a comment saying they spend more on a SINGLE HOUR of a dev's time than the monthly GH bill, which is not true for an org of more than 20 people or so (depending on hourly rate).

[kleebeesh]

Ah, totally misread it. Thanks.

[zeku]

Just a bone to pick... new grad engineers in my US city started around 60-70k in 2018 when my college cohort graduated. Southern US...

[zrail]

Things have changed considerably over the last four years.

[tjoff]

Well, considering you'd likely spend an average of 5 minutes per day doing it I wouldn't mind it.

[oceanplexian]

There are a lot of problems with this from the business angle:

(1) An engineer getting paid 120k doesn't "cost" 120k, probably >150k with federal taxes, health insurance, benefits, and so on. Not including the cost to recruit, interview, and train said person.

(2) I don't know of many 1,000 person companies that would trust a new grad software engineer with no experience to manage critical infrastructure.

(3) You need N engineers to manage said service, because what happens when your one engineer gets sick, takes PTO, or quits for some reason? You also need a manager for said engineer(s).

(4) You now need to secure an internal service you never did before, so expect to have to hire external security consultants or re-allocate security engineers, since it's high risk.

(5) Github is FedRAMP compliant, SOC1 and SOC2 compliant and GDPR compliant. If you or your customers need any of those things, expect to hire external auditors on a recurring basis to validate your home-grown solution meets those requirements.

I hate to make these points because I'm a big believer in the scrappy startup mentality, but if you want to do things right, in the context of a large enterprise that is accountable to a lot of people, expect a project like this to cost $1MM per year minimum, and it probably won't reach parity with a cloud offering in terms of reliability, multi-region performance, proper backups, and so on. This is why Github can charge ~$200 per user (Or $200k per year for 1,000 seats) and still come away looking like a bargain.

[xondono]

I’d say it depends, I run my own on prem server and gitlab was a PITA. Too many moving parts, updating took too much of my time, and I never felt “safe”.

Moving to gitea solved all of those issues for me (thus far), now I’m looking into adding other stuff like CI through Drone.

[Inversechi]

Did you consider woodpecker instead of drone? It's basically an evolved fork of the OSS version.

https://woodpecker-ci.org/

[xondono]

Didn’t even know about it. I’ll check it out.

Thanks!

[KronisLV]

Curiously, this was also my own experience!

I actually wrote a bit about the migration process, as well as the reasons for migrating over to Gitea, Nexus and Drone CI as opposed to using GitLab, GitLab Registry and GitLab CI: https://blog.kronis.dev/articles/goodbye-gitlab-hello-gitea-...

With containers, it's actually a pretty good experience that's not too hard to setup or manage.

[edgyquant]

It definitely depends. We’re pretty early stage and I’m the senior engineer+infrastructure guy so running our own gitea instance or whatever is just more time that I’m almost out of.

[jeltz]

Maybe you are in a different time zone because our organization certainly noticed and was disrupted by this.

[edgyquant]

I’m on PST time, some of our other devs are on the east coast and one is in India. I think we’re spread out enough it should be an issue but maybe we prioritize different things.

[jeltz]

We are in CET and maybe we use Github differently than you.

[jhugo]

I think the impact was for some reason not consistent between users (maybe due to geographical factors or maybe sharding of accounts?). We're in Asia and I think we've had three different days recently where we couldn't actually get much work done due to GitHub being flakey or down for the entire day and our CI/CD and development processes being built around it. We ended up moving off GitHub onto a self-hosted system, which took about a day of work for one engineer (CI/CD itself was already self-hosted, so just Git, issues and PRs), and there have already been two more GitHub outages since then.

[jmartens]

My company monitors the functionality, performance and availability of apps like Github, and we have certainly noticed the increase in issues lately.

[edgyquant]

We were actually talking about implementing this last week. Not for GitHub but for slack as it seems to have issues once a month or so.

[ManWith2Plans]

I will say that for us this is a huge deal. We're a devops services company, and our customers expect their deployment pipelines to work. This is becoming a huge pain-point for a few of our customers and we recommended Github Actions to them. A couple of our customers want us to move away from GitHub actions because of how disruptive outages have been.

[jacobr]

20 PRs waiting in line for half a day to be merged is pretty annoying. We’ve had that on multiple occasions the last few weeks due to GitHub incidents.