|
|
|
|
|
|
by nostrademons
5613 days ago
|
|
|
Be careful extrapolating success scaling in one domain to success scaling in another. I've also done the 40 GB/day of NYSE TAQ data financial analysis thing, and the 1000+ trades/second real-time financial analytics thing. And I work on Google Search, and have a passing familiarity with how other Google products scale. The scaling challenges of batch financial models vs. real-time financial processing vs. information retrieval vs. email vs. social products are very different. Even going from a model of the web where it's static and changes every few months (like Google of 2004) to one where sites get update every few minutes and users expect to see the updates immediately in search results (like Google of today) requires vastly different technology. The main thing about scaling that I've learned from working at a couple places that require it is to go into it with a fresh mind each time, and really pay attention to what the requirements are and what you can cut corners on. There're some general principles you should know (eg. Jeff Dean's "Numbers you should know", memory is much faster than disk, cut out layers of abstraction that you don't need), but in order to apply them effectively, you really need to pay attention to the details of your problem domain. If you think you can solve Twitter's scaling problems, they're hiring, they're pre-IPO, and they're probably giving out decent chunks of stock. |
|
|
I agree about that.
> go into it with a fresh mind each time, and really pay attention to what the requirements are and what you can cut corners on. There are some general principles you should know: memory is much faster than disk, cut out layers of abstraction that you don't need, etc - but in order to apply them effectively, you really need to pay attention to the details of your problem domain.
(slightly edited) - This is golden.
However:
> If you think you can solve Twitter's scaling problems, they're hiring, they're pre-IPO, and they're probably giving out decent chunks of stock.
I know I can solve Twitter's scaling problems (I don't think the solution I posted is the end-all-be-all, and for all I know that might not be where their scale problem is -- it is just perceived and argued about this part, which is not very hard).
However, Twitter's abysmal uptime (for the kind of sevice they are providing) had no bearing on their growth in 2008-2009. And even if by re-architecting Twitter they can save $2M/year on operations, it would be dumb to do that before they're in the black for a while and can identify their real profit and loss centers.
Also, those stock are not worth quite as much as people think when you take everything into account. (I've got a successful exit as a non-founder behind me; I'm intimately familiar with all the gory details including taxes, dilution, etc -- Whether options or RSUs, if you are granted anything of value, you have to pay full taxes AT THE TIME OF THE GRANT).
My point was only to show how non-impressive the problem twitter is (supposedly) facing. It's a repeating discussion:
Don't know why I even bother anymore. A company that had (maybe still has?) their millions-of-views-a-day pages created in Ruby doesn't care about solving scale issues.At Google, you guys throw out closing paragraph tags from the main page when it is clear it renders fine without them.