[calcorbin]

I'm realizing I've only worked in distributed systems as well, but I'd never feel comfortable telling potential employers I'm an expert. Being an expert in distributed systems seems almost too broad. At a high level couldn't it be expertise at integration, accessible logging, and configuration?

[0xbadcafebee]

Distributed systems research has been going on since the 70's and Unix Neckbeards have probably forgotten more about them than we have learned, so actually I think impostor syndrome is a bit warranted with them.

The actual hard stuff is not even these papers, it's the implementations that are way more complex than some algorithm or architectural pattern. Anyone who says "X is better than Y" is fooling themselves because it's only the implementation context that matters.

The only thing you can say for certain is that reducing the amount of components and complexity in the system often results in better outcomes.

[webmaven]

> The only thing you can say for certain is that reducing the amount of components and complexity in the system often results in better outcomes.

No, there are a few other things that you can say for certain:

Watch out for positive-only feedback loops, you absolutely need negative feedback as well - or only. Eg. exponential back-off.

Sometimes, you just need a decentralized solution, rather than a distributed one, and you don't have to have the same answer at every scale (eg. distributed intra-datacenter, decentralized inter-datacenter, or vice-versa).

Loose coupling is your friend.

Sure, add an extra layer of indirection, but you probably need to pay more attention to cache invalidation than you think.

Throughput probably matters more than latency.

Reducing the size/number of writes will probably help more than trying to speed them up.

Multi-tenancy is a PITA for systems in general, and distributed ones are no exception (aside: there is probably a huge business for multi-tenancy-as-a-service, if anyone manages to solve it in a general-purpose way), but a series of per-customer single-tenant deployments may be worse, especially if they are all on different versions of the code. Here be dragons.

Don't overthink it. Start with a naive implementation and go from there (see loose coupling above).

[0xbadcafebee]

Maybe there are some other things you can say for certain. But as to some of your points:

> Watch out for positive-only feedback loops, you absolutely need negative feedback as well - or only. Eg. exponential back-off.

Agreed it may need negative feedback, but I'm not sure about always.

If your service has a latency SLA, exponential back-off might kill your SLA (depending on wordage and where the back-off is). The fix is to soft reject requests (RST rather than dropping packets) when you can't meet the demand. This change may allow you to meet your SLA if it's written to prioritize low latency over service unavailability.

This is it's own negative feedback loop, but change from sending RSTs to silently dropping and you no longer have the feedback.

> Sometimes, you just need a decentralized solution, rather than a distributed one

Agreed

> Loose coupling is your friend.

Until it isn't? :)

> add an extra layer of indirection, but you probably need to pay more attention to cache invalidation

Fixes for additional layers tend to increase system complexity compared to fixes for fewer layers.

> Throughput probably matters more than latency.

Until it doesn't :)

> Reducing the size/number of writes will probably help more than trying to speed them up.

Depending on 20 different things... You really have to account for all the system's limits (and business use cases) and find the solution that matches the implementation needs.

> there is probably a huge business for multi-tenancy-as-a-service

Sure, it's called EKS :-) Just build more clusters... Don't worry, we'll bill you...

> Don't overthink it

Yes and no; Yes, in that there will always be unknowns. But no, in that often improvements in communication will provide better solutions without extra work. Think smarter, not harder!

[webmaven]

Every maxim has caveats and exceptions. Including this one.

[ystad]

Yes. The area is fairly broad. In my opinion - I think the question to ask for is: Do you have the distributed systems mindset? Not Are you an expert on Distributed Systems

[shred45]

It is definitely a broad term, and I think that disciplined implementation of the things that you mentioned is the real key. It just isn't as exciting to talk about.