[SEJeff]

People often misunderstand the difference between failover, high availability, and load balancing.

Failover - n +1 node waiting to take over in the event of the primary node falling over

High availability - n + 2 - requires a minimum of 3 nodes to decide "quorum" and "elect" a master node. This often involves hardware level fencing and STONITH (google the name if you're not aware).

Load Balancing - distributing load amongst multiple nodes to scale horizontally better.

In a perfect world, you are using a data store that supports master / master replication. Then you just front your data store with a load balancer like haproxy. You can ensure the load balancer stays up by running multiple of them on different nodes with setting up up failover with something like keepalived. Sucks this happened at DO, but perhaps it will help you build more robust infrastructure in the future.

Always design for each individual component to fail.

[coolboykl]

So in this case, do I needs to setup two nodes for HAProxy as well, to load balance MYSQL Cluster?

[SEJeff]

Yes, and then setup keepalived on them so you have a stable VIP that is guaranteed to move between them. I did a writeup of how we did it (when I worked there) for ticketmaster here:

http://www.digitalprognosis.com/opensource/scripts/keepalive...

The gist is that you have a dummy interface ie: dummy0, and when you down that interface, the VIP flips to the backup node with the highest VRRP priority.