[somehnacct3757]

As the person who gets pinged when our CI isn't working, I'm hoping my next employer doesn't have github in their stack.

We've reached the stage where architectural decisions are being questioned. Nobody should need to ask me "well why are we using github?" but here we are.

They needed to fix this two weeks ago. The next time I'm in charge of stack decisions I will be evaluating competitors. This is exactly how Slack beat Hipchat.

Don't let weekends dilute your view of the situation. There's only 23 weekdays in March and Github has not been reliable for 5 of them.

[ho_schi]

Host on premise? Or use Gitlab as service or - again - on premise. Outsourcing in general and literally "outsourcing" should be done for things you cannot do yourself properly (no competence, too much work, no feasible resources).

Long read but describing it neatly: https://danluu.com/nothing-works/

A decision between - do you pay your personnel for doing it - or - do you give others money and hope they do it in a way which is good enough. Outsourcing as trend failed because it was driven by capitalism isn't a new finding. And as convenient removte web services are, the cloud is outsourcing.

[jameshush]

I'm curious, what's stopping you from hosting on-premise and/or using your own runners?

[jrowley]

Using your own runners GitHub doesn’t completely solve the problem, because GitHub won’t queue them to start the job during these outages.

[somehnacct3757]

We aren't a big enough team. The resources to do such a thing would mean less time focused on customers.

Maybe after another doubling we will have these options available. We should be a perfect fit at this stage for a hosted solution. Github is making themselves unviable for a company at this stage (low $XX million ARR)

[BeefWellington]

"We aren't big enough to self-host" is an example of the effectiveness of SaaS sales pitches.

Don't get me wrong, there's a lot of good reasons to go cloud.

There's not a lot of good reasons for a build system to be completely down with no backup plans when your SaaS provider has an outage.

What does even just Github Actions' availability look like for March? 90%? That's pretty brutal for what should be a five nines service.

[somehnacct3757]

We have other tools that are self hosted and they require regular maintenance to security patch, write storage retention policies when some stupid log fills up, have backups, migrate cloud images, stop using plugin with deprecated api, etc. This stuff can also be unplanned and even scheduled becomes a regular chore taking time away from customers.

I know the on-prem sharks smell blood in the water but on-prem is not all upside and not the default answer to hosted services having downtime.

For example we could be on another DevOps SaaS and have zero downtime in March.

[redisman]

We can run the CI by hand or move it to another platform. But it takes some time and effort. This uptime is abysmal and has already stalled my work days twice in the last few weeks so maybe it’s worth the change