I'm a self-taught, have a single dedicated server, I have a single-instance Kafka running on top of ZK. Yes, I lose the benefits of replication, failover, etc. I don't need that though. The whole installation took me half an hour, learning Kafka took maybe 3 hours, and as long as my server's been up, Kafka's been up.
Granted, I am not monitoring Kafka, but I do other processes.
The other nice thing is that now that I have ZK, other software that need it can just reuse the same process.
I think using the maintenance cost as a reason to write your own tool, is a short-sighted decision.
Even if that's the case, deploying and scaling a Kafka cluster is something that hundreds of companies have figured out and publicly written about. It's something that you can hire an experienced engineer to fix. When this thing runs into problems, they will be all new ones.
> I love the concepts Kafka defines so clearly, but the software is too complex and have dozens of knobs you have to adjust.
This is one of our biggest headaches, and it's not even that a Kafka server itself is so configurable. We have hundreds of teams writing client applications, and jumping on bridges because Kafka clients have poor configurations is getting old. Too many knobs to twiddle, but I guess that's what happens if you're expecting to be able to tweak for high performance.
Granted, I am not monitoring Kafka, but I do other processes.
The other nice thing is that now that I have ZK, other software that need it can just reuse the same process.
I think using the maintenance cost as a reason to write your own tool, is a short-sighted decision.