Word count of 1.13B messages
- Storm: ~16K QPS/node, 100ms per event (P999)
- Spark Streaming: 100K QPS/node, 1s batch window
- Concord: 500K QPS/node, 10ms per event (P999)
Server log processing (29G server log, ~260M msgs)
7 different computations including deduplication, counting, pattern matching, windowing...
4 nodes, 8 vCPU, 32GB RAM each
Concord: 1M – 1.8M QPS / cluster
Spark Streaming: 72K – 2M QPS / cluster
Concord generally performed in the consistent range of 1-1.8M QPS for whereas Spark's throughput varied differently based on window sliding / amount of internal shared state.
Word count of 1.13B messages - Storm: ~16K QPS/node, 100ms per event (P999) - Spark Streaming: 100K QPS/node, 1s batch window - Concord: 500K QPS/node, 10ms per event (P999)
Server log processing (29G server log, ~260M msgs) 7 different computations including deduplication, counting, pattern matching, windowing... 4 nodes, 8 vCPU, 32GB RAM each Concord: 1M – 1.8M QPS / cluster Spark Streaming: 72K – 2M QPS / cluster
Concord generally performed in the consistent range of 1-1.8M QPS for whereas Spark's throughput varied differently based on window sliding / amount of internal shared state.