| > the definition of the MapReduce pattern? Impala, Presto etc don't fit that model at all - they follow the Volcano model. In this mode, they are not pure functional - if a task fails, there is no way to reproduce the output of that task. The func within the map() was guaranteed to produce the same output for the same input across multiple attempts on failure or concurrently (for speculation). Because of this, they can be faster as they do not wait for a task to be complete to run a subsequent stage & can pipeline better, but at the cost of failing all queries running on a node during a crash. There are no retries for anything. This was deemed acceptable, if your hardware is reliable and the response to a failed query is just to "run it again", rather than per-node query recovery. The reason for proper node failure tolerance for Spark/Tez/Flink etc are because they follow the functional model as closely as possible with exceptions for non-deterministic functions (say, UUID() in a SQL call). The advantage of the failure tolerance is that these tools can push the whole cluster towards a single query performance when it is otherwise idle, because preemption can recover capacity out of a running system, if a higher priority query enters the system at a later point in time. |