[domano]

For some serivces in a kafka-based system i replaced the Postgres some services were using with File-based BoltDB stores or SQLite DBs.

With kafka as the data source every instance was autonomous and would keep itself up to date using the kafka partition offsets.

Went really well, took the DB cluster out of the scaling equasion and offloaded a lot of the cost to the SSD instead of memory or a managed DB Cluster.

If eventual consistency is enough i feel that this is a nice way to handle things.

[eb0la]

Kafka is works very well when you have a IF "receive-this" then that kind of problem.

I find a bit complicated using streams / queues for everything because it is a paradigm shift (turning the database inside out).

BUT when you have a clear use case like ingestion -> refinement step 1-> refinement step 2 it is a _very_ natural way to do stuff with the added benefit I can have as many parallel workers as topic partitions.