[drinker]

Those are all good points and pros for redpanda vs Kafka but my question stills stands. Isn't redpanda designed for high-volume scale similar to the use cases for Kafka rather than the low volume workloads talked about in the article?

[rockwotj]

When the founder started it was designed to be two things:

* easy to use * more efficient and lower latency than the big resources needed for Kafka

The efficiency really matters at scale and low latency yes but the simplicity of deployment and use is also a huge win.

[munksbeer]

In kafka, if you require the highest durability for messages, you configure multiple nodes on different hosts, and probably data centres, and you require acks=all. I'd say this is the thing that pushes latency up, rather than the code execution of kafka itself.

How does redpanda compare under those constraints?

[rockwotj]

Oh if you care about durability on Kafka vs Redpanda, see https://www.redpanda.com/blog/why-fsync-is-needed-for-data-s..., acks=all does not fsync (by default before acknowledging the write), so it's still not safe. We use raft for the data path, a proven replication protocol (not the custom ISR protocol) and fsync by default for safety (although if you're good with relaxed durability like in Kafka you can enable that too: https://www.redpanda.com/blog/write-caching-performance-benc...).

As for Redpanda vs Kafka in multi AZ setups and latency, the big win in Redpanda is tail latencies are kept low (we have a variety of techniques to do this). Here's some numbers here: https://www.redpanda.com/blog/kafka-kraft-vs-redpanda-perfor...

Multi AZ latency is mostly single digit millisecond (ref: https://www.bitsand.cloud/posts/cross-az-latencies/) and the JVM can easily take just as long during GC, which can drive up those tail latencies.

[enether]

It's pretty safe. Kafka replicates to 3 nodes (no fsync) before the request is completed. What are the odds of all 3 nodes (running in different data centers) failing at the same time?

[rockwotj]

Generally if you care about safety then “pretty safe” doesn’t cut it.

[enether]

It's just my polite way of saying it's safe enough for most use cases and that you're wrong.

The fsync thing is complete FUD by RedPanda. They later introduce write caching[1] and call it an innovation[2]. I notice you also work for them.

Nevertheless, those that are super concerned with safety usually run with an RF of 5 (e.g banks). And you can configure Kafka to fsync as often as you want[3]

1 - https://www.redpanda.com/blog/write-caching-performance-benc... 2 - https://www.linkedin.com/posts/timfox_oh-you-cant-make-this-... 3 - https://kafka.apache.org/documentation/#brokerconfigs_log.fl...