Experience with messaging platforms. Keep in mind that Twitters platform is not particularly high volume for messaging systems. It is reasonably high volume for messaging between people. But people tweet rarely enough for Twitters overall volume to be all that special.
And Twitters core message exchange functionality is conceptually extremely simple.
You need to be able to take a message, and put it into the set of targets, T.
Problem is T can be "unreasonably big". E.g. millions of followers. Solution? If something is too big, see if you can divide the problem. Split T into smaller chunks t1..tn, and distribute them. If those chunks are still too large, split them again.
This reduces your problem to figuring out how to insert and remove and balance a tree, and how to process two kinds of message deliveries: Insert into timeline or reflect message to specified set of targets (each of which can be another reflector, or a final target timeline)
We all know of and use a system that does this at massive scale: E-mail w/forwarding and mailing lists. If you really wanted to (though it'd be extremely inefficient) you can do this with off the shelf mail-servers and other e-mail software. Using mail servers as messaging middleware might sound weird (and there are certainly better alternatives today), but it works just fine.
At Twitter scale there's certainly enough savings to be made from figuring various obvious and not so obvious optimizations (off the top of my head: for some of the users with the really extremely follower counts, it may make sense not to push their messages, but to push them to caches and weave them in dynamically) to justify the salaries of a few software developers to cut down on hosting and ops costs, but the basic problem is not hard.
He's actually dead on. After that it's a 'mere matter of programming', so there's plenty of work to be done but that's the exact way how you tackle this particular problem.
The "handwaving" here are simple, well understood problems both on the development side and operational side. As I've mentioned elsewhere: Twitters message volume or messaging patterns are nothing special. And it's a problem that is trivial to subdivide into manageable chunks.
And Twitters core message exchange functionality is conceptually extremely simple.
You need to be able to take a message, and put it into the set of targets, T.
Problem is T can be "unreasonably big". E.g. millions of followers. Solution? If something is too big, see if you can divide the problem. Split T into smaller chunks t1..tn, and distribute them. If those chunks are still too large, split them again.
This reduces your problem to figuring out how to insert and remove and balance a tree, and how to process two kinds of message deliveries: Insert into timeline or reflect message to specified set of targets (each of which can be another reflector, or a final target timeline)
We all know of and use a system that does this at massive scale: E-mail w/forwarding and mailing lists. If you really wanted to (though it'd be extremely inefficient) you can do this with off the shelf mail-servers and other e-mail software. Using mail servers as messaging middleware might sound weird (and there are certainly better alternatives today), but it works just fine.
At Twitter scale there's certainly enough savings to be made from figuring various obvious and not so obvious optimizations (off the top of my head: for some of the users with the really extremely follower counts, it may make sense not to push their messages, but to push them to caches and weave them in dynamically) to justify the salaries of a few software developers to cut down on hosting and ops costs, but the basic problem is not hard.