[dasil003]

Give me a break, scaling Twitter is not easy. CS principles do not magically allow you to construct an optimal solution for real world engineering problems. At Twitter's volume, the tiniest bit of extra latency in some middleware or network hardware can be the bottleneck.

It's so easy to sit here and play armchair engineer and diagnose Twitter's (mostly past, btw) problems with a nauseating mix of hubris and ignorance.

[Retric]

What makes people laugh at twitter is the problem is not that hard, but they make all the classic mistakes.

Every time someone wrote a reply Twitter... this was the most expensive work the database

Sorry, I am going to laugh at people having scaling issues when they use a database like that on a messaging system. There was an essay by PG where as soon as he heard they where using Oracle (or C?) he instantly knew to ignore them for much the same reason.

[tectonic]

And yet, Twitter is kind of hard to ignore.

[gaius]

So was Pets.com.

[pg]

Would you please not resort to personal attacks?

[dasil003]

I'll try, but let me ask you this, which post was more personally offensive? What I said in response to one flip comment? Or what the comment said about a whole group of successful people without any reasoning or substantive argument?

[pg]

Yours was worse. His was a type of inane throwaway remark that's at least marginally acceptable to make about famous groups or individuals. E.g. "the 49ers suck this year." Yours was a bitter, personal attack on an individual.

[serhei]

> ... with a nauseating mix of hubris and ignorance.

That, and the pointing out of quite natural questions which Twitter has failed to address in its explanation of how it is acting. I mean, once you know what you are doing, being able to satisfy people asking ignorant and hubristic questions is a pretty sweet bonus.

[rjurney]

Scaling twitter would have been relatively easy using Erlang, because it has all the hard parts solved as basic features of the language. They decided to go fishing instead.

Anyone doing this over without a point to prove would start with http://www.rabbitmq.com/

[a-priori]

Erlang is not a silver bullet. It would not magically solve all of Twitter's woes. Neither would RabbitMQ for that matter.

[rjurney]

Its not a silver bullet, but it is specifically designed for problems like Twitter's back-end issues. If it can handle running telephone systems for years at a time with a large string of 9s of uptime, I think it can handle broadcasting SMS. Well. Its the most proven tool for this type of job and the only one I'm aware of designed from the ground up for building applications like twitter that are utterly reliable.

Not a case of magic, just the right tool for the right job. You could code a web app in C, but why would you? Making twitter in Ruby (and to a lesser extent Scala) is the same thing.

I clicked the profile link to your blog, and it looks like you have a lot of Ruby experience. Compare the syntax for message passing between Erlang and Ruby.

id ! {my_fun,Args},

Thats it. Passing messages between pids on nodes across a network isn't much harder.

Compare the difficulty of parallel computation and storing data in parallel across many systems reliably between Erlang and Ruby. Doing that stuff in Erlang is just as easy as doing CRUD in RAILS, because the tools are designed to make it easy.

The downside is that Erlang is pretty bizarre and hard to learn if you've never coded in a functional language - you only get to set values ONCE, etc. Thats a critical feature for debugging large pools of processes working together, but it sure is strange. Which is why you wouldn't use Erlang to do a simple CRUD web app, you'd use RAILS.

Seriously... its an entire language based around passing messages between lightweight pids working in congress, across many systems, with built-in fail-safes for any failures, and a rock-solid distributed/redundant data store baked in.

In this situation its just about as close to a silver bullet as you will ever see. Erlang would make writing twitter's back-end FUN.

A dozen crazy Swedish mad scientists slaved over a language for years to enable creating scalable applications like twitter trivial, only to be totally ignored by most everyone until quite recently. And still twitter picks something else for the rebuild.

'Heavy metals poisoning' would seem to be a more apt metaphor than 'silver bullet.'

[gaius]

nauseating mix of hubris and ignorance

I'm afraid it's a nauseating mix of having operated real pub-sub systems such as TIBCO Rendezvous on major stock exchanges.

What Twitter does is not difficult and they do it badly.

[joshu]

I haven't been in the industry since 2005 or so. Even so:

And how many stock symbols are there? 5500 on NYSE, 4500 on NASDAQ.

Updates per second? CSCO had a few hundred thousand updates per day.

How is the data persisted? It is typically only logged, and not retrieved in realtime.

Unfortunately, Twitter doesn't match this problem particularly well.

Delicious, which I did write, had many similar structural issues. Tags, /network, etc.

[ubernostrum]

So now you bait and switch?

You started out with a condescending post which implied that anyone who'd stayed awake in CS 101 would find Twitter trivially easy to scale.

Then you switched course to claiming that you know how to do this because of experience working a gigantic installation.

Or are you asserting that the average CS 101 project consists of running the production messaging systems for major stock exchanges?

[ellyagg]

That wasn't a bait and switch. He suggested the engineering problem isn't as hard as is being made out, and then when you claimed otherwise, he explained that he was in a strong position to know otherwise. In the real world, this is known as a "good argument".

He didn't claim his experience proved HE could do it. He didn't imply that you NEEDED his experience to do it. He was calling you out on your claim by showing that he had done it, and so knew from experience that it wasn't hard. You're the one moving the goal posts.

Note: I actually don't have an opinion on whether that experience is relevant to twitter's situation, but it certainly wasn't a bait and switch.

[ubernostrum]

I didn't claim otherwise; you'll note that that was my first comment in that little thread.

The fact that he went from "CS 101" to "major stock exchange" as the barometer seemed a little fishy to me, ya know?

[gaius]

Evidently the concept of studying theory in school then applying that theory in the practical world of industry is too complicated to fit into 140 chars.

[dasil003]

How much did Twitter cost to develop vs TIBCO Rendezvous? How much of the "easy" part did you come up with yourself vs working at a polished product that already? What are the actual volumes that Rendezvous handles and how do the details of the network topography compare? How do the details of the functionality compare? How much scalability analysis to they teach in CS (answer: only theory)? Is anyone who doesn't have your particular experience an idiot? Is working on a product for a small collection of the filthy rich better work than creating something that millions of people directly identify with?

Finally, how much do you know about Twitter's architecture and do you know the definition of hubris?

[gaius]

How much did Twitter cost to develop vs TIBCO Rendezvous?

The real question is how much VC has Twitter wasted reinventing the wheel (and making it square) when they could have just bought an R/V license?

"Not invented here" is the definition of hubris.