Hacker News new | ask | show | jobs
by sitkack 1613 days ago
I am trying to be more positive in general, take everything with a grain of salt, I also work for a Big Cloud provider.

I read that as we work really hard to engineer crystalline fault lines vertically through our stack so the system has a nice clean single plane of fracture.

Given their track record of reliability and the unsubstantiated claims in the article, I can't even. In the real world, all the actions that have absolutely saved a system was an occurrence of fallback.

Having branch free code, one way to fail is nice from a reasoning perspective, and reasoning was more than one of the points brought up in the article. But reasoning is a goal that is different than reliability. I can use a reliable automatic transmission without reasoning about it.

Fallback fixes issues that failover doesn't. Rather put out a piece that encourages someone to not do something (sometimes this is important granted), encouraging folks to use immutability would be a larger global positive.

Immutability really does change everything.

https://cacm.acm.org/magazines/2016/1/195722-immutability-ch...

2 comments

I mean, I can definitely see their point. I work in distributed systems for a decade and I can tell you, when you kick the can downstream, it just gets worse later when it’s spread out and systemic.

You should nip overloads in the bud, and not propagate them. Have backpressure be at the protocol level, and every node only deals with its neighbors.

In fact, I would go so far as to say that the main reason for these failures is because we have monolitic, global addressing systems like DNS or IP routing tables, which let me send spam email to anyone, or DDOS a site from many machines at once. It’s totally discontinuous.

What a good distributed system should have is be continuous in distributing capabilities. Each node can grant capabilities only to trusted neighbors, and revoke any that have been misused. Neighbors can then delegate some capabilities to others, or — if the node wants — forward an invitation to them, to become a neighbor.

That would also solve all the issues about “real names policy”, and other crap like that. It shouldn’t matter whether you are “the real” Bill Gates or not. Your email shouldn’t be accessible to the whole world.

And websites would also be stored using a FileCoin-type market, which recruits more machines as more readers SPEND MONEY using micropayments to access the files.

Right now micropayments aren’t feasible, so instead we essentially have the publishers pay for hosting and collect micropayments via subscriptions and bundles.

Immutability doesn't really solve everything. It provides a cleaner path for retries for writes, but still doesn't handle situations where reads fail.

I think the conclusion in the article ("don't do fallback") is misguided. Fallback code is sketchy, but sometimes it is worth it to take the time to write well-audited, well-tested fallback code to ensure a system which has high availability requirements can survive dependencies which are less reliable.

So we agree 100%! We should talk more.