[gfodor]

Ok, it's clear now you have horrible reading comprehension and/or have never used Hadoop in practice. If you perform an aggregation on 10 fields in a log file, a common pattern, you are going to be generating more intermediate data than your log files during the shuffle. (Hadoop is, in this case.) If you generate a dataset that has pairwise scores for users, even if you are only sampling 1% of them than you are generating way more data than your input. (You are generating this data in this example, not Hadoop, which is why I included it.) Say N=1M. You want to compute the top 1% pairwise scores (pruned via some locality sensitive hashing.) Guess what, you are computing N^2 * 0.01 = 10B records. Perfectly tractable for a big data system. You seem too busy yelling at me to even read what I am writing, since I stated this clearly: "(I realize in practice you generally do not compute the entire space because it will be too slow. However your output will be closer to n^2 than n, ie, it will be much larger than your input.)"

You say: "These are benefits of EC2, not of map-reduce. The critique was specific to map-reduce, not of the entire principle of on-demand computing." This is patently false. The criticism of the author is that people using EMR are a cargo cult and do not understand the purpose of map reduce. Ie, they should not be using EMR, and he is so clever for realizing the folly of their ways. My claim is the author does not understand the purpose of EMR. By definition, EMR brings with it the benefits of EC2. I don't see how this is a strawman.

At least I am sure now that it's not my communication skills, I illustrated two perfectly fine examples and you glossed right over them without understanding. I've already spent enough time on this and argued in good faith and at this point I can at least be confident that you have no interest in understanding what I am saying and instead want to childishly prove me wrong on the internet.