[Swinx43]

So much this! As someone that regularly deals with data platforms and data engineering issues this nails it.

Today’s desire to make everything event driven and “real-time” boggles the mind. Solutions are contorted to fit the Kafka operating model.

The inevitable result is blog posts about how “we solved our self inflicted problems that would have never existed if we chose a sane design for our real problem. “

Oh and never forget that no senior stake holder ever really checks their info/dashboards in real time. Never mind actually making decisions in real time based on it.

[dominotw]

Same! My dataengineering team wants everything to be moved from batch oriented to 'stream oriented'. We are replacing our batch workflows to flink/spark streaming. But no one has been able to answer me what business use cases need streaming or what if any problem is being solved by this added level of complexity.

[disgruntledphd2]

Sometimes, streaming makes sense. When I was at a FAANG, we had a sampled streaming database with nice plots (kinda, ugly but functional).

This was incredibly useful as it allowed teams to figure out what had happened in prod without needing to wait for the next day's batch jobs.

OTOH, most companies don't need something this complicated, and I'd imagine you could do the same with MySQL/Postgres if you're not at FAANG scale.

[Swinx43]

I agree. When it is the right solution to the problem at hand there is nothing better than a good streaming solution.

It goes for pretty much all technology, that you need to choose the right tool for the job. For some reason the marketing just gets most people all worked up and ready to fit problems into technology that was never designed for their problems.