Hacker News new | ask | show | jobs
by xapata 1045 days ago
How is coordinating between different machines any different than coordinating between different processes?

A multiprocessing implementation is a good prototype for a distributed implementation.

1 comments

To be honest, I don't think both are similar at all.

Parallelizing across machines involves networks, and well, that's why we have jepsen, and byzantine failures, and eventual consistency, and net splits, and leadership election, and discovery - so in short a stack of hard problems that in and of itself is usually much larger than what you're trying to solve with multiprocessing.

True, the networking causes trouble. I usually rely on a communication layer that addresses those troubles. A good message queue makes the two paradigms quite similar. Or something like Dask (https://www.dask.org/). Having your single-machine development environment able to reproduce nearly all the bugs that arise in production is a wonderful thing.