[bkirwi]

Oh hi!

The Samza ecosystem seems to be a magnet for this sort of thing these days -- the community's really committed to taking full advantage of Kafka, and not just papering over it so it looks just like any other queuing system. Really excited to see where all this will lead.

[Terr_]

Speaking as a bystander who mainly reads blogs, my perception of Kafka is that it's marketing (for lack of a better term) has successfully positioned it with "we can keep your arbitrarily large logs" and "they are pretty durable, consumers can revisit them if they fall behind".

This offers a nice distinction from MQ-ish systems, where the emphasis seems to be on a different set of benefits like "we can handle complex centralized distribution logic for you" and "we help manage your synchronous calls".

Does that seem accurate in terms of how Kafka is evolving its future niche? Personally, I'm interested in optimistic-locking when adding to a log.

[bkirwi]

Yes, Kafka certainly is good at those things, and I suspect Kafka will only get better at them. But it's actually quite a simple bit of infrastructure at heart[0] and this means it's useful for a large variety of other things, many of which we're only figuring out now.

Re: optimistic locking... there are a couple proposals for this floating around, and I'm not sure which / if one will get in. It certainly seems consistent with the general mission, though.

[0] http://engineering.linkedin.com/distributed-systems/log-what...