[ransom1538]

I have a few serious concerns: Suppose have nodes A,B,C,D running with slaves and you have a user u1 start retrieving and storing data. The odds will be high that the user u1 stores keys on all nodes A,B,C,D since all the keys are based on a numerical hashed slot.

1) The user u1 will end up connecting to all nodes. Thus as you scale adding an "E" you always end up connecting to each cluster having the same amount of connections. Example, let's say A,B,C,D have 5k connections open and you add "E". Now "E" has 5k connections. Now you are blocked by the number of requests per second and no amount of hardware can save you.

2) Let's say B and B1 die. (It happens.) Now your system is completely out. All 5k connections are blocked. It would be nice if A,C,D,E continue to run.

I usually get around things like this using an index server between user u1 and A,B,C,D. When u1 starts retrieving keys the system needs to be designed such that u1 only connects to single node for it's keys.

How do you get around 1,2? Do other people use index servers?

[antirez]

Hello, I'm not sure I totally get 1 and 2, maybe sometimes you write "cluster" instead of "node"? Btw I'll try to reply.

1) The amount of connections is not very relevant. You can set a max limit if you want, like with the redis-rb-client that will close old connections before opening new ones if the limit set by the user is less than the number of master nodes. However this is not recommendable as actually what is an idle connection? Just an entry in a data structure of the kernel. So it is not an issue for every client running in a given box to have multiple connections to multiple nodes.

2) The limit in the partition resistance in Redis Cluster is a fundamental design trade off, it means no need for a "merge" operation and that in a given moment only one node is responsible for a given key. I think that in the case of Redis most of the times less partition tolerance in favour of a simpler consistency model is what users need.

EDIT: a clarification on "1", what I mean is that every connection will be active only for the subset of keys it holds, so basically the traffic of every client is split across all its connections. Similarly every node of the cluster only performs a percentage of work proportional to the number of hash slots it holds compared to the full set of 16k hash slots.

EDIT 2: it is not clear from your point "2" if you understand you can have B2, B3, B4, BN if you want.

[seiji]

such that u1 only connects to single node for it's keys. Do other people use index servers?

The redis cluster client in your programming language will know the entire current redis cluster state. The redis cluster client will automatically place (and read) keys directly to (and from) the correct instances. The direct reading also reduces latency found in some other cluster implementations where you only have the option to proxy your requests. You as an application programmer never have to worry about multiple servers.

When your redis cluster client connects to the cluster, the redis client reads the current key slot to server map, and uses that for every action (with updates as necessary as failures/promotions happen).

Let's say B and B1 die.

Good example, but you should probably be running with at a minimum B, B1, and B2 (and three entire copies of A, C, D as well). You have to design your deployments in concert with how your underlying architecture works. Redis cluster instances do not have to take up an entire machine. Run a couple masters and a couple replicas per host. Running a cluster with 4 masters means you should run with 8 replicas (minimum). See the math section of https://github.com/mattsta/redis-cluster-playground for more examples.

It's worth stepping back and defining "Cluster" too. A "database cluster" has no meaning by itself. It doesn't specify redundancy, how availability works, or how replication works. A Riak cluster has nothing to do with a Redis cluster. You have to learn the operations on each type as you go.

[antirez]

About that, redis-trib, the Redis Cluster config utility, while still pretty basic is already able to allocate masters and slaves in different IPs, so if you give it six nodes like 10.0.0.1:6379 10.0.0.1:6380 10.0.0.2:6379 10.0.0.2:6380 10.0.0.3:6379 10.0.0.3:6380, you can expect a master for each IP address, and replicas always allocated in nodes where there is NOT its master.

[mbell]

In general I think you're right but I'm not sure why you would have 5,000 connections to each node. There isn't any reason I can think of to maintain more than 1 connection per client (application process) to each node. You don't gain anything by 'connection pooling' with Redis so it's not like each app server would want to keep a 100 connection pool around. I guess you could have an issue if you have 5000 application processes all wanting to connect to a single redis cluster, but I'd argue that you'd be well past the point of needing to break your system into services by then anyway.