[saryant]

I've run Kafka at large scale. I've also seen even larger scale attempts to replace it.

Just use Kafka. Seriously, it's rock solid and is practically lingua franca in backend architecture these days. Everyone understands it and every data processing framework or service supports it.

Kafka is much, much more than just distributed pub/sub. It's disk cache optimizations alone make rolling your own a terrible idea.

[kylecordes]

This is a principle our industry implements poorly. It often seems like each new generation (for very small values of the word generation) must reinvent the same thing. Perhaps because the old thing was too complex to understand immediately... complexity driven by the needs of the underlying problem... complexity which the new implementation will inevitably obtain if it survives long enough and becomes popular enough that anyone cares.

(That said, I'm highly in favor of innovation of most any kind; building new things is great, if the new thing has some plausible innovation over the old thing!)

[borplk]

The industry itself feeds this cycle by rewarding people for creating new projects and demanding shiny github projects.

[maltalex]

It’s true not just in software , but in many aspects of modern society. We value innovation (even when it is bot innovative) a lot more than maintenance. Just look at physical infrastructure as an example.

[bonesss]

> Kafka is much, much more than just distributed pub/sub

In between 0% Kafka and 100% Kafka: Kafkas REST proxy gives a minimalistic API surface for using the system. Recreating that API in another language with a simpler backend is highly achievable...

My Kafka installation has a parallel relational backend that provides an onboarding story for smaller apps and groups, for example. It provides about 13% of Kafkas functionality and can't scale meaningfully, but provides baseline data streaming in a pinch and is API compatible with how we use Kafka most of the time.