| Hi HN, A few weeks ago, we shared GlassFlow: Open Source streaming ETL to dedup and join streams from Kafka for ClickHouse (https://news.ycombinator.com/item?id=43953722). One of the top questions we received was: “How well does it perform at high throughput?” We ran a load test and would like to share some results with you. Summary of the test: - Tested on 20m records - Kafka produced 55,000 records/sec - Processing rate of GlassFlow (deduplication): 9,000+ records/sec - Measured on a MacBook Pro (M3 Max) - End-to-end latency: <0.12 ms per request Here is the blog post with full test results and tried with different parameters (rps, # of publishers, etc.):
https://www.glassflow.dev/blog/load-test-glass-flow-for-clic... It was important to us to set up the testing in a way that everybody could reproduce. Here are the docs:
https://docs.glassflow.dev/load-test/setup We would love to get feedback, especially from folks consuming high-throughput in ClickHouse. Thanks for reading! Ashish and Armend (founders) |
Everything was running on the same machine?