[hodgesrm]

Thanks, this is a great article. The money quote for me is:

> I think it makes more sense to think of your datacenter as a giant database, in that database Kafka is the commit log, and these various storage systems are kinds of derived indexes or views.

My company supports analytic systems and we see this pattern constantly. It's also sort of a Pat Helland view of the world that subsumes a large fraction of data management under a relatively simple idea. [1] What's interesting is that Pat also sees it as a way to avoid sticky coordination problems as well.

[1] http://cidrdb.org/cidr2015/Papers/CIDR15_Paper16.pdf

[skyde]

The problem is that Kafka API and Commit log API are very different.

If you wanted to literally use Kafka for your commit log the same way the Amazon aurora are using a distributed commit log. You would find that a lot of feature a commit log need are missing and impossible to add to kafka.

[arduanika]

Greenspun's 11th law: Any sufficiently stateful program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of SQL / relational databases.

[weddpros]

is that another definition for confirmation bias?

I can name so many use cases a relational database is incapable of...

[Aqueous]

Yes. as the article points out there's no way to reject an event based on some criteria that keeps the events internally consistent. for instance, two people can't check out the same item of inventory = 1. you need to first validate this event on the materialized view of the commit log to make sure the item is still available. however, what happens when your validation goes out of date? For example, what if there's an event checking the last inventory in the event log but the materialized view doesn't reflect that yet? you end up checking out the same inventory twice and promising it to both customers! not good.

DBs can solve this through optimistic locking (atomically making sure my new event applies to the version number of the previous event + 1, otherwise failing) but there's no way to do this in Kafka as far as I know (however, as this is a problem i'm currently facing, please feel free to let me know if there is)

[hinkley]

I’d really love for this to be formalized.a standard format streaming the data through several Susie,s and keeping them eventually consistent.