[mrbungie]

Well, it's an engineering decision, so there is no direct answer.

But as an engineering manager, at least I must ask and answer the following question: even when nobody wants _slow_ data, how fast is fast enough? I don't see decision makers choosing and thinking better with a 10 min latency vs 20 min latency, as they are not looking at the reports all the time, even for big events like Black Friday (they have meetings and stuff you know, even their supporting analyst teams do).

For more time-critical matters (i.e. real time BI or real time automatic microdecision making for fraud detection), as I said, we did have the capability to run both more frequent microbatches or do RT processing using Flink connected directly to our app backend messaging system (ironically Confluent, Kafka as a Service). But that is very different to using a complex real time log as Kafka running on "pet" servers as the cornerstone of your data platform and then propagating said data to different engines/datastores (at least 4 as I said) for downstream processing. That's a lot of moving parts running in a low reliability environment.

Overengineering is a thing, and I think it was my responsability at the time to limit the level of complexity considering the reality of the business and the resources we had in the team, even if that meant 20 minutes of latency for a business report. That's my point and why I say I think it was a bad decision to use a Kafka based stack. YMMV obviously.