[nathas]

Nice write-up from Marc. This definitely hits on the most common problems distributed systems face. I haven't read the paper yet but it is pretty cool they published this and talk about changes over time.

1. Managing 'heat' in the system (or assuming that you'll have an uniform distribution of requests)

2. Recovering a distributed system from a cold state and what that implies for your caches.

3. The obvious one that people that do this type of thing spend a lot of time thinking about: CAP theorem shenanigans and using Paxos.

Reminds me of the Grugbrained developer on microservices: https://grugbrain.dev/#grug-on-microservices

Good luck getting every piece working on the first major recovery. My 100% unscientific hunch is that most folks aren't testing their cold state recovery from a big failure, much how folks don't test their database restoration solutions (or historically haven't).