[otterley]

Let me ask the question in reverse: what do you have against them such that the fellow human beings struggling to maintain their operations don’t deserve even a modicum of kindness, respect, and good will? Are you unable to separate the business from the hard working people behind it?

It’s not like they don’t know that people like us are counting on them: they recognize that their service is the “dial tone” for much of the world’s software development capability. They are keenly aware of the impact.

What happened to #hugops? Does it go out the window because those people happen to work for a company you don’t like?

[StableAlkyne]

When did OP blame the people involved personally?

If I to hire a contractor to redo my roof, and that roof leaks, whether they worked hard or not is immaterial. They did not do the task in they were paid to do. I'm not going to buy their services again just because their shingles guy was particularly charming.

MS has talented engineers, but that's a complete misdirection. Github is a service in decline: there is nothing wrong with criticizing them.

[ebiester]

I have all the empathy for people in the world.

A corporation is not a person. If your organization cannot handle the load, then you need to adjust your practices. The organization needs to prioritize their paying users. The organization needs to shift people from new features to keeping the lights on. And maybe the organization needs to find another strategy to manage its azure transition.

[otterley]

A corporation is made of people. GitHub cannot exist but for the people who continue to work for it. And they’ve already said, multiple times, that restoring availability is their top priority.

[port11]

A corporation is made of people, but its ethos is the product of decision-making. If a corporation is consistently, say, unethical, is it because they hire only unethical individuals? Or because unethical people somewhere along the chain of command make unethical decisions?

[otterley]

I'm not exactly sure what you're getting at with this question. It seems to still conflate corporate-level decisions with boots-on-the-ground work.

Are you suggesting that whatever decisions their upper-level management makes that you consider unethical irreversibly and irrevocably taints all the difficult and honorable work that their engineers and operations people are performing?

[port11]

I’m saying their lower-level employees are probably honest, hard-working people like everyone else. But the detachment that comes from a large corporate structure makes the higher-ups decide things that aren’t as honourable.

“Corporations are made up of people” is a strange way to excuse the reality that the ‘bad’ things that corporations do are often decided by top management.

[otterley]

Ah. I didn’t intend to excuse the decisions of upper management when I said that. My intent was to counter the notion that a corporation and its workers can’t be analyzed independently.

A corporation is just a business formation, and businesses are made of individual people working for it. Those people’s motivations and efforts can, and often should, be evaluated separately from the decisions of management.

[logicchains]

>What happened to #hugops? Does it go out the window because those people happen to work for a company you don’t like?

Would you feel the same way about a colleague who kept causing downtime in your product again and again, seemingly without making any progress in addressing whatever issue was causing their repeated mistakes?

There are web applications out there that are far more complex than GitHub but have much less downtime. It's not like they're facing an unsolvable problem.

[otterley]

You don’t know that it was “their mistake.” Unless you’ve personally successfully scaled a suite of nontrivial services equivalent to GitHub’s to accommodate an unexpected 14x increase in traffic, you respectfully have no basis for such an assertion.

[dijit]

I have.

You could argue the scales are different, but computers are also faster now.

So, argument to credentialism out of the way... What should we do as consumers if a provider that is a defacto monopoly due to network effects stops functioning?

https://news.ycombinator.com/item?id=47947719

https://www.linkedin.com/in/jharasym/

[otterley]

> You could argue the scales are different, but computers are also faster now.

Scale is everything and a faster computer doesn’t always help. Vertical scaling has limits, and complex distributed systems are complex.

Since you seem to possess a diagnosis and remedy with a reasonable amount of certainty, I’m sure they’d love to hear from you and have you fix all their problems for them. Especially if you can do it while not making the problem worse in any dimension.

[dijit]

The link in my previous comment answers the credentials question in detail- including specific technical post-mortems on horizontally scaled stateful systems. Vertical scaling wasn't the topic.

[otterley]

You’re missing the point: a doctor doesn’t diagnose and practice medicine on a patient he hasn’t thoroughly evaluated himself. This is the sort of wisdom that a staff engineer and CTO is expected to have earned.

[Aurornis]

> I have.

I skimmed your profile. Working on the infrastructure for a couple mid-tier video games is a cool accomplishment, but equating this to having solved GitHub level scale rings hollow.

GitHub has a couple orders of magnitude more daily active visitors than the games you worked on had at their peak.

You can make valid criticisms of GitHub without trying to reduce their scale or inflate your credentials to create a false equivalence.

[dijit]

"false equivalence" needs an equivalence claim to be false.

I didn't make one. The sentence after "I have" was literally "you could argue the scales are different."

GitHub spent a decade asking the world to host its code with them. They got what they asked for. You don't get to beg everyone to run services for you for ten years and then have "scaling is hard" be the answer. They should be improving, not regressing over time, and they have some of the worlds best engineers and a trillion dollar corporation behind them, they don't need my sympathy.

The original question is still open and nobody's engaging with it.

[Aurornis]

> I didn't make one. The sentence after "I have" was literally "you could argue the scales are different."

Don't you at least see how it's misleading to respond "I have" in response to a question about scaling GitHub-scale services?

Trying to caveat it with "the scales are different" misses the point. The parent commenter was talking about scale.

[cnewey]

I'm not sure that resorting to personal attacks against the parent commenter for making a legitimate critique is the right, fair, sensible, or mature approach here.

Discarding legitimate criticism based on some self-determined criteria of intellectual superiority isn't a good look. It smacks of elitism and isn't something conducive to a productive and positive community discussion.

It is unhelpful, rude, condescending, and completely fails to address the underlying problem.

[otterley]

The commenter inserted his own personal bona fides (as a proxy for skill, experience, and knowledge) and use them to bolster his conclusion of culpability and incompetence of the GitHub team. If you take that risk, you should expect to be challenged if those skills are not up to par.

Put more simply: if you get into the ring, you’d better be prepared to take a punch.

[Aurornis]

Not a personal attack to fact check someone's claims.

I didn't bring their credentials into the conversation. They did.

[VirusNewbie]

Ok, well, I work on systems quite a bit larger than Github, and I think they have a major reliability issue.

[otterley]

That’s not in dispute. The question is whether we should be supportive of the company’s efforts to improve reliability, or whether we should keep punching down. How would you feel if you were in a similar situation and outsiders breathlessly provided uninformed opinions about your problem and questioned your competence?

It’s all about the Golden Rule.

[mattmanser]

Yeah, they should be testing for that, right? I think there's a lot of people reading comments like yours and thinking, is this person a paid shill or what?

The earn bucket loads of money, they should be planning for exactly that. And testing for it via load testing every day.

Perhaps you've forgotten the days of GitHub presenting themselves of software engineering thought leaders.

[otterley]

I’ve worked at some very well-endowed organizations. Having money is no guarantee of a particular outcome. There is a lot of money chasing a limited supply of talent. Moreover, distributed systems that were built long ago with certain assumptions can’t be refactored as quickly as the HN populace might believe. The Mythical Man-Month is a popular book for a reason.

[c-hendricks]

> Perhaps you've forgotten the days of GitHub presenting themselves of software engineering thought leaders

Genuinely could use a refresher here.

[nimih]

I think it's possible to be simultaneously: gracious and supportive towards the developers and ops staff who have been struggling to maintain reasonable uptime on the extremely important piece of shared internet infrastructure that everyone commenting probably relies on (either directly or indirectly) on a daily basis; and spiteful and cruel towards the massive (and, historically speaking, ethically fraught) corporation whose cynical acquisition and subsequent mismanagement of that same resource got us here in the first place.

[otterley]

I agree 100%! But this important distinction and nuance seems to be lost here.

[ofjcihen]

OP didn’t blame the staff. His focus is on the company.

Invoking individual workers well-being to defend a billion dollar company is also very strange.

[otterley]

A company is made of individual workers. That doesn’t change because there are a lot of them or that their employer has a lot of money.

[ofjcihen]

If anything your argument against MS’s uniqueness makes OPs case stronger.

[otterley]

Tell us how.

[VirusNewbie]

Executives have made a choice to not pay for top talent at Microsoft Azure and Github.

[otterley]

Would you consider telling this to the people working at GitHub directly? I’m sure they’d appreciate your evaluation of their skills and talent.

[falcor84]

There are two options, either they are lousy at their jobs, or they are incapable of pushing back against unrealistic demands. Neither is a good indicator of their skill and talent as engineers.

I know I am speaking from a position of some privilege, but I have previously left workplaces that did not allow me to practice good engineering, and I do expect others to do so.

[fragmede]

Or, they've been given crap primitives to work with. There's only so much lipstick you can put on a pig. I don't know what database they're using or what their pub sub and streaming looks like, or even what their system diagram actually looks like. But, well, you don't see Google having these kinds of problems. Other ones, sure, but between Chubby and Spanner, if Google had bought GitHub we wouldn't be having these problems.

[falcor84]

But it wasn't a pig. It was a reliable system, and then it increasingly became an unreliable one, in a way that is not explainable by the mere increase in demand. Whatever rearchitecture was performed, it was done and is apparently being perpetuated by software engineers who should be held accountable. Not necessarily guilty, or even directly at fault, but accountable nevertheless. "I am just an employee of a bad company" is not a valid excuse for an engineer.

[VirusNewbie]

eh...

https://github.blog/news-insights/company-news/oct21-post-in...

They tried to scale MySql and turn it into Cassandra and then they lost customer data (that they claimed to later recover).

[vablings]

In a SWE job market like this, do you really want to be seen as the "conscientious objector"?

There are literally thousands of people who are ready to ride up the totem pole, it would not be a difficult decision for a bad manager to swing his axe and replace the new head

[einsteinx2]

Talented engineers shouldn’t have much problem finding another position even in this market (of course they should find one before leaving I’m not discounting family responsibilities and whatnot), so if your argument is they’re not able to leave and find another job then you’re essentially agreeing with the person you’re replying to.

[otterley]

Really? Only two possibilities?

[VirusNewbie]

yes I would tell them "you're underpaid, if you can, come to a company that appreciates your talents more".

[otterley]

What you said came across as an adverse judgment of their skill and talent. Is that not what you meant?

[VirusNewbie]

I've interviewed at many large tech companies including Microsoft, and Microsoft Azure was mostly a clown show. So, yes I judge the talent there. I'm sure there are some superstars that I'd love to work with, maybe I'd even work there and try to fix it for the right price, but god damn was it a stark contrast to other companies.

[tardedmeme]

Are you hiring?

[VirusNewbie]

Yes google cloud is actively hiring.

[turtlebits]

#hugops is to your coworkers, not to the nameless big-corps who can't maintain a service for paying customers. You should be raising a shitstorm when things you pay for aren't reliable or unusable.

Hot take, if it's traffic is causing issues, throttle your free-tier, pause signups, or stop giving out free things (like runner time).

[otterley]

Who is “maintain[ing] the service”? The workers, of course!

[estimator7292]

If you pay someone full price to do a job, they know they can't fulfil the terms up front, accept the work, deliver less than the agreed upon terms and still charge you full price, you'd probably call that transaction fraudulent.

GitHub is promising service they know they cannot meet, not telling you that, and still charging you full price. What's more, one can argue quite convincingly that they're lying about their level of delivered service by not reflecting the actual level of uptime on their status page.

To give benefit of the doubt requires that the other party is not blatantly and overtly acting in bad faith. When they are, you're just apologizing for fraudulent behavior.

[otterley]

Fraud is a serious civil and criminal accusation that’s not to be taken or given lightly. Can you detail the fraud that’s being committed? What is the specific promise they made that you’re being deprived of? Remember, the four corners of your agreement with them are controlling.