[rewmie]

> 8000 messages a day, tops? That’s 5 a minute. Does that warrant “infrastructure”? I think a gameboy’s Z80 could handle that load.

The blog post is quite clear in stating that their pain points had nothing to do with scaling or throughput. The author explicitly mentions idempotency, custom headers, and authentication.

I think you're ranting about a strawman you put up.

[ImPostingOnHN]

As someone who helps administer several kafka clusters which carry millions of messages per second, and works with both DBs and DBAs, the shallowness of the author's complaints seemed to belie a lack of experience and understanding of either technology. If they gave even a small amount of detail, it might have been useful:

"We couldn’t create a Kafka source connector from the database holding the messages" – Why couldn't they?

"we had extreme difficulty troubleshooting when their system refused our best understanding of said custom headers" – What made this so extremely difficult for them?

"Authentication with Kafka, and authentication to encrypt confidential data in Kafka, and getting the Avro schema from the schema registry were all separate and painful steps" – These are 3 totally unrelated features, so of course they each have settings. What made the steps so painful for them?

"We would never tolerate this level of complexity with a database" the author says of configuring kafka features, apparently missing that a database setup with all those features (external authentication, encryption, custom serialization) also requires "separate" configuration for each one.

[karmakaze]

It seems to me that Kafka was the strawman that the article put up to take down as an antipattern. There was no part of the premise that made sense.

> While my client was in a situation where we had no choice but to use this monstrosity of Kafka, Avro, and custom message headers, I would never recommend this usage of Kafka if I had the option.

The antipattern isn't Kafka, it's the client situation.

[jshen]

The author's talked about their pain points, but they didn't really outline the downside of going to this approach in a comprehensive way. What I've found with event based systems is that they are much worse if your operational maturity/excellence is low. To make it worse, operational excellence can get worse over the years so you can't simply base the decision on how well you do it today.

[SanderNL]

I think you might be right. I’ll take the hit.

Still, I think talking about Kafka without load feels like Christmas without tree. I thought that was the main point of it. Handling massive loads (by distributing them).

[rewmie]

I think that the author just leveraged the existing messaging infrastructure they were already using for other services, and also reused their know-how. Their new microservice will barely register in the overall traffic volume, but they don't need to either deploy dedicated infrastructure or onboard onto yet another technology just to have message listeners.