[tetha]

This is what I've been starting to think about on a more abstract level: Introducing a new technology, a new system into a design isn't like putting a piece into a jigsaw puzzle, or even worse, trying to mold and force the system to fit whatever hole your design has. Many more specialized systems - and Kafka is one of them - should solve some problem, but they should also change your mental model of the system and you should look for the easiest way to introduce these heavy hitters.

For example, if you use Kafka or streaming solutions like Flink or Spark, you should change your mental model to (possibly large), (possibly resplayable) streams of events and look for simple ways to get these event streams going and good ways to consume them. And then you need to let the design push you where it wants you to go.

Like, at work, we recently had a discussion how it was so storage-expensive for a project to store all events of a day and how the query to count all of these events per tenant was taking so long. While they are using a streaming event processor in front of it. Like, what the hell - think in streams, tally up these events on the fly and persist that every hour?