[Confiks]

Is there anyone here running Kafka in production as primary data store for an web application backend? Did you write a very custom integration (or indeed a ES architecture), or did it just fit into existing (MVC) frameworks?

I appreciated the mid-article note that "treating Kafka as a primary store does mean raising the bar for how you run it". Databases and filesystems are silently scary.

[papaf]

I imagine that you have to use an Event Sourcing architecture because if you use Kafka as a primary datastore you need to add a lot of complexity outside of Kafka to make it usable.

For instance, you cannot query Kafka and this is a basic requirement of most data stores. To get queryable data, you could build up state from events in some external component and then query that external component.

From what I understand, this building of state from events is an example of an Event Sourcing architecture.

I think event sourcing as a primary datastore is a cool idea. Its fun to play with such ideas on small projects and then scale up. The downside is that Kafka is a distributed monster that is hard to run locally (you need multiple docker containers for a basic setup). I'd be very interested to see if anybody else answers your question.

[hactually]

Yeah, i had to mess around heaps to get it working nicely on my Mac. Hope this helps someone!

https://gist.github.com/17twenty/d619f922ab209f750630824f7c6...

[Confiks]

Okay, so, what about the New York Times:

https://www.confluent.io/blog/publishing-apache-kafka-new-yo...

[manigandham]

As the first comment [1] says, it's a silly architecture for them and completely unnecessary.

All modern databases are essentially the same: they receive changes, save them to a write-ahead log or journal, then materialize the data into indexes and tables. This is the event-sourcing + compaction + materialized views model that kafka/confluent keeps pushing but inside a single piece of software that does all the hard things for you with arbitrary queries, reliable backups, etc. There's a reason why this is the common model for decades, there's nothing new here.

Splitting up all that logic into separate event streams, processing pipelines and datastores is certainly valid but only really necessary at the 1% of companies that operate at that scale. For most people, a relational database running on a mid-range server is plenty of power and far better long-term storage.

1. https://news.ycombinator.com/item?id=15184640