[manigandham]

Kafka and Redis are very different things - see this: https://news.ycombinator.com/item?id=11577312

Redis is a database, Kakfa is a data logging system built for scale and throughput. Event processing (of any kind like stocks, ad impressions, ecommerce purchases) are a great fit. Also good as a message queue unless you need ultra low-latency RPC.

[nodesocket]

Gotcha, so then advantage of Kafka over Logstash + ElasticSearch?

[saryant]

Kafka, Redis and Logstash+ElasticSearch have really nothing to do with each other.

Kafka is a distributed, fault-tolerant and highly scalable message broker.

Redis is a very fast key/value (another other data types) store.

I suppose that at a high level Logstash can be compared to Kafka but IME Logstash can't handle scale. It's trivially easy to bring Logstash to its knees.

Elasticsearch is, well, a search engine.

[010a]

Redis does a lot more than just store keys and values. Functionally speaking Redis Pub/Sub and Kafka are interchangable up to a certain level of throughput.

[saryant]

They really serve very different use cases. Redis pub/sub consumers only receive messages while connected, whereas Kafka consumers can pick up where they left off. Event ordering, back pressure, etc.

There are many use cases Redis pub/sub can't serve beyond just scalability.

[balamaci]

Well Logstash can output data into Kafka or ElasticSearch. So you could for example transform logs to json or do simple text processing in Logstash and put it to ElasticSearch for logsearching but you can also put it in Kafka and then have a stream parsing with all sorts of tools like Flink, Spark, etc. You could then have the possibility do so some realtime analysis on what the user do all over your stack. Too many "Login Failed" events and maybe you have an attacker trying to bruteforce a passsword and maybe you need to present him with a captcha screen.

[manigandham]

ElasticSearch is a database optimized for searching, not related at all.

You can somewhat compare Kafka to Logstash but Kafka has no processing, it's purely a distributed log writing/reading/storage system that also scales far more than logstash can. You write data to it and then read from it with a basic messaging abstraction of topics and partitions.

[ec109685]

ElasticSearch can store sequenced number data, which is really all that Kafka is doing, so I don't think it is fair to say it isn't related at all.

[allengeorge]

So can a RDBMS... But that doesn't mean that Kafka and databases are related.

As multiple comments have stated above, Kafka is really a distributed message subsystem. Its core interface is a set of topics that one can publish to, and that consumers can read from (in other words, a pub-sub system). Kafka doesn't inspect the message payload at all.

Elasticsearch is a unstructured (to some extent) document store that's optimized around document search. So at the very least, the payload is important when using Elasticsearch.

[manigandham]

That's like saying they all write data to disk so they're related.

Elasticsearch is all about saving, inspecting, indexing and retrieving your data through a rich document-based model and search-optimized methods.

ES might be able to do the same thing functionally because it operates at a higher level but ultimately will never scale or be as simple in access as Kafka.