[MontyCarloHall]

Assuming the chains can reach an equilibrium point (i.e. burn in) quickly, M samples from an MCMC can be parallelized by running N chains in parallel each for M/N iterations. You still end up with M total samples from your target distribution.

You’re only out of luck if each iteration is too compute intense to fit on one worker node, even if each iteration might be embarrassingly parallelizable, since the overhead of having to aggregate computations across workers at every iteration would be too high.

[itissid]

In reality the number of chains is not that many though, right? I've seen 3-4 chains in models in STAN that can do the job on most smallish(econ, social sciences) datasets, though I maybe wrong about other domains...