[victork2]

Agreed, but sadly this is the state of passing information now in "new age" startup/ companies.

It's like Twitter: some nice pastel colors, very little text, no information but it's easy access and people can say "Oh yeah, I know about scaling I have read this on a blog post". Sorry to break it everyone but scaling is hard, totally unfunny or uncool, with tons of problems and stressful.

But to answer your question ans scaling is for most companies kept secret because it is such a key process that they don't really want to share it with their competition. If you find some I would be interested too, I am currently in the process of doing so in my company and it's a pain.

Anyway that put me in a bad mood :).

edit: you really shouldn't be downvoted for stating a cold truth

[phillmv]

1) It's a slide deck for a talk. What were you expecting, a detailed tutorial? I also don't think you got the most important bit: focus on employees.

2) Scaling is also… really custom. I'm extremely unlikely to have to find a way to shard git repositories across a network of users. And they do cover their architecture and release back a fair number of their tools.

[ans]

Re.: 1) OK-why post it then? Post the talk.

Re.: 2) Doesn't matter if it might not be useful, it sure would be interesting. :)

[pjhyett]

We've gone into excruciating detail about how GitHub's architected: https://github.com/blog/530-how-we-made-github-fast

[Pewpewarrows]

Sorry to break it to you, but scaling isn't hard anymore. The tools we have at our disposal today makes it trivial compared to companies attempting to do the same thing 5-6 years ago. As soon as you figure out the scaling from 1 -> 2 servers, you're basically done until you hit top-1000 traffic. Because as soon as you spend a weekend doing that, scaling to 2 -> N is child's play.

Oh look, I'll pick from any number of battle-tested, high performance, and free load balancers. I'll spin up a couple more VPS nodes. I'll add a few more slaves to my DB setup. I'll install some memcached and a quick write-through wrapper in my code. Oh yeah, duh, don't do long-running operations on the request/response cycle, here's an open source queue system for you. And since I like going overboard, let's throw some Varnish all over the place. Congrats, we've just covered the scaling concerns of 95% of web apps.

Try doing that several years ago with the terrible PHP and Java services we were writing, and without most of the hardened tools we now get from thirty seconds of Googling and learning from everyone else's mistakes.

Disclaimer: yes, when you start to break through the top-1000 some hard problems start appearing again. You begin to run into the limits of what others have created, and patching / rolling your own solutions becomes commonplace.

[victork2]

Well first I said I was in a bad mood.

More seriously, scaling is not simple. Yes products have been introduced or have evolved and simplified greatly the task. But saying that scaling is simple is simply not true. Not everybody do simple web apps, some require very heavy processing behind the scene, Hadoop/ MapReduce operations that have to return complex results in memory constrained environment and all that while maintaining low cost. As an example when Relational database are not sufficient anymore and ACID is not casting its reassuring shadow you can be in trouble if you do things like payroll/ real time process etc...

And again you don't have to be in the top 1000, many companies do not even have a lot of traffic from the general public but have to deal with very hard scaling problem.

Just look at Reddit, they are still having trouble with the Cassandra database which is said to be easiest to scale.

My 5 cents on the question!

[getsat]

Reddit is a top 200 site. They do over two billion pageviews/month. A lot of their pages are loading 1,000+ nested replies. They use Postgres as their primary datastore.

Scaling is a solved problem until you're huge if you're not doing something really avante-garde.

[adgar]

> A lot of their pages are loading 1,000+ nested replies.

Reddit only shows 200 comments by default. It also used to be you needed Reddit gold to even show 1,000 on the page when it first loads, not sure if you still do (my gold expired)

[adgar]

> A lot of their pages are loading 1,000+ nested replies.

Reddit only shows 200 comments by default. It also used to be you needed Reddit gold to even show 1,000 on the page when it first loads.

[alttag]

Okay, you've made the point that scaling technology is easy ... what is less easy (for example) is scaling the business processes to support in increased number of users or handling the communication overhead of an up-sized development team. People-centric processes cause significant scaling trouble.

Yes, aspects of scaling may be easier, but scaling (in general) is still a complex beast that kills many companies.