[dhaivatpandya]

Communication would become a bottleneck regardless of whether you're using MPI or something else. The problem you're talking about has nothing to do with MPI; it is intrinsic to distributed computing itself.