I believe this is referring to the probability concept of "Independent and Identically distributed".
In the usage in the book/page, it seems to refer to how tasks/problems are run in parallel and the learning averaged, whereas the author is advocating these problems run sequentially.