Hacker News new | ask | show | jobs
by hashmp 3560 days ago
Use 5 zookeepers, on a separate set of servers.

Use configuration management such as chef to allow you to quickly build new nodes and to roll out changes accross the cluster. You will need to make tweaks. The chef Kafka cookbook which is the top result on Google has means of coordinating restarts of brokers accross the cluster. Use consul as a locking mechanism for this. You could use zookeeper, but consul works well for auto DNS registration and auto discovery.

Use the yahoo Kafka-manager app to manage the cluster and to see what is going on.

Don't use the Kafka default of storing data in /tmp/. Your OS will periodically clean it.
3 comments
Gigablah 3560 days ago
> Don't use the Kafka default of storing data in /tmp/.

That seems like MySQL level of bad defaults.

link
cbsmith 3559 days ago
Five kind of kills performance compared to three, and doesn't map well into AWS, where you generally have 3 or 4 AZ's. I tend to go with three but make sure you've got fully automated responses towards failures.
link
ZenoArrow 3560 days ago
Five zookeepers? Seems like a lot. Why five? Is it hard to keep them active?

Thanks for the tips.

link
darkr 3560 days ago
Zab [the distributed consensus algorithm that powers ZK] shares some similarities with Paxos, and requires a quorum of nodes to be online.

If you want highly available ZK, your choices are 3, 5, 7... nodes, for which you can have 1, 2, or 3 nodes offline at any one time.

If you have one node fully down on a 3 node cluster, and there is even a tiny network blip or partition (as often happens in cloud environments) then you are down.

link