[jcims]

I've worked at two large companies now with a mature managed Kafka offerings. The 'platform' engineering team handles all of the engineering, implementation, security and compliance, upgrades, observability etc. and have self-service onboarding with lots of recipes and sample integrations. My team moves about 5B messages a day through two topics and we're not putting a dent in the overall volume. It just enables use to move so much more quickly than we would if we had to deal with all of that ourselves.

So in our case it's clearly not an anti-pattern, but the right tool for the job.

[atomicnumber3]

The fact that you had an entire team operating it is the key, I think. I've seen various big buzzwordy techs used at various shops I've worked at and whether it was nice to work with totally came down to whether there was a team behind it operating it for us.

K8s with a k8s team running it - fabulous. Without a team and everyone kind of just needs to know enough to get by, except for the one guy who set it up? Dreadful.

Airflow when there's a team running it? Great. Luigi when it's just you, another dude, and the one guy who set it up? Not great.

Even RDS is like eh. Still had RDS tip over and we had to do manual vacuuming or something with compacting tables with dead tuples. Annoying when we were paying for RDS to ostensibly not have to do this.

[browningstreet]

Agreed -- have used it in a bank. It was very suited for that.