[rbranson]

How is this scalable? It seems like this is a single-instance solution that in theory could scale, but this isn't even proven, nor is a clear path given. The author throws out a few vague ideas, and seems to vastly underestimate the complexity of scaling out. Redis cluster also is nowhere near stable, and keeps getting pushed back, so how does this help anyone now?

[fritzy]

This is not a single instance solution. We can run as many instances as we want across multiple servers. It is currently a single-Redis instance solution, however we have some intelligent sharding planned.

I agree with you about Redis clustering, and would extend it by saying I don't think it'd be helpful for this problem regardless. Intelligent sharding with some gossip for slave promotion is where we'll have to go.

See my follow-up post tomorrow.

[rbranson]

Yes, single Redis, but multiple instances of the application backed with a single datastore is hardly revolutionary. This is the classic reason why people go with a stateless app layer, so that it's simple to scale at least a single tier of the architecture.

I'm still not sure how this lives up to the "scalable realtime architecture" that dominates the title. People have been building Redis-backed message brokers for a while now. It seems like you have some untested ideas for "intelligent sharding" that could possibly provide scalability, but that's punting on the hard problem. Without this, this is all really pretty pedestrian.

The gossiping sounds like an interesting lead, is this going to be discussed in the post tomorrow?

[fritzy]

Evolutionary, and probably not even a unique idea. My post does go into the intelligent sharding a bit. Essentially sharding, in the case of &bang, on teams with a simple lookup to find which server a given team is on. Allowing the stateless app layer (as you rightfully call it, other than keeping track of subscription event routing) to connect to multiple Redis servers as needed. You could be on two teams on separate servers and it would work fine.

I also agree that people have been using Redis for messaging, however, what we're doing is a step past that. Again, probably others are doing this. We have atomic verbs for dealing with higher level objects in Redis that use Redis-PubSub for communicating the changes to the objects. This way, whenever an object (feed) is changed (publish/edit/delete/reposition), it can bubble up to the user or processes that care. These feeds are broken down by not only topic but subscribe-able units of interest as well. In this way, processes, users, etc only get updates on data that is relevant to them.

I'm working on converting these verbs to Redis-Lua scripts as my tests have shown it decreases CPU time and reduces atomicity code (especially in Node.js where watch->multi->exec callback stacks can be interrupted by other events). I also expect it to make supporting Thoonk in multiple languages easier as the core code will be shared.

The gossiping is currently being discussed in the Redis mailing list and may end up taking a similar approach to Hadoop's intelligent clients. Antirez would like to provide some tools to make this easier as others have rolled their own Redis gossiping. I'll be researching that more after implementing the team sharding. HA, while related to scaling, is a bit off topic, and I admit to hand-waving here. For now we're focusing more on not losing/corrupting user data.

Perhaps Henrik's title is hyperbole. I doubt the approaches that we're taking are revolutionary, and I imagine similar things been done before. However, we believe that it is a good approach, and we believe in the direction we're taking enough to share it.

Thanks for the intelligent discussion.

[rbranson]

Cool, thanks for taking the time to go a bit more in depth on this. This is a topic I'm particularly interested in, so I'm looking forward to the follow-up.

[fritzy]

To clarify, by "Intelligent Sharding" I meant application specific sharding to keep objects together that are related to each other.