[lifefeed]

I was interviewing for software jobs recently, and while I was studying up on the "system design" portion I kept circling around the same insight that Dan Luu writes about so well here.

I would sit down at an interview and try to create these "proper" system designs with boxes and arrows and failovers and caches and well tuned databases. But in the back of my mind I kept thinking, "didn't Facebook scale to a billion users with PHP, MySQL, and Memcache?"

It reminds me of "Command-line Tools can be 235x Faster than your Hadoop Cluster" at https://adamdrake.com/command-line-tools-can-be-235x-faster-... , and the occasional post by https://rachelbythebay.com/w/ where she builds a box that's just fast and with very basic tooling (and a lot of know-how).

[hinkley]

Before 'eventual consistency' was coined as a phrase, there was an old, powerful and deeply unsexy form of eventual consistency called "batch processing".

For small batches you do an interval at a convenient time, such as a time of day where the hardware is undersubscribed for some other task. As the batches grow then you have an online system that continues to work until you get rather far down the list of consequences in queuing theory. Once you get 24 hours and 1 minute of tasks per day you never catch up (it never ceases to amaze me how often I can find someone who will fight me on this point), and you must be aware that substantially before that breaking point, you can experience rather long average queuing delays.

But if your workload is spiky, you can smear 250 minutes of peak traffic out over 6-18 hours with no problems at all. You need a safe place to stash the queue and a little sophistication around recovering from failures/upgrades. Those aren't necessarily Simple tools, but if that's the most complex part of your system you're doing pretty okay.

[mapme]

I think that’s a large oversimplification of Facebook. While it’s true a lot of FB storage is MySQL backed they also created many complex systems such as:

- Cassandra (based on dynamo/big table)

- wrote a custom KV store named RocksDb that is open source/now a company

- wrote a custom photos storage system that replaced an NFS based design

- wrote another custom binary object store

- wrote a custom geo distributed graph db (Tao)

- wrote an in house distributed FS replacement for HDFS

https://www.cs.cornell.edu/projects/ladis2009/papers/lakshma...

https://www.usenix.org/legacy/event/osdi10/tech/full_papers/...

https://www.usenix.org/system/files/conference/osdi14/osdi14...

https://www.usenix.org/system/files/conference/atc13/atc13-b...

https://www.cs.princeton.edu/~wlloyd/papers/tectonic-fast21....

https://m.facebook.com/nt/screen/?params=%7B%22note_id%22%3A...

[kragen]

I think all of this was after their first billion users?

[dismantlethesun]

Cassandra was released in 2008. Facebook hit 1 billion users ~ 2012, and 2 billion ~ 2017. Back when Cassandra was released, they had a 'mere' 100 million users.

Facebook stats:

https://www.statista.com/statistics/264810/number-of-monthly...

[kragen]

Thanks for the correction! So the more correct statement would be that Facebook scaled to 100 million users with PHP, MySQL, and Memcache, and then to a billion users with Cassandra and Haystack, and then built all the other stuff after their first billion?

[mapme]

Yeah, there’s also a lag sometimes ie. They write the paper 1-2 years after building the system. They’ve published some really interesting papers over time. If you filter by year you can see them all!

https://research.facebook.com/publications/

[stackbutterflow]

Some people say that Leetcode is nothing more than rote learning. Some disagree and I disagree. There's a minimal amount of rote learning that helps but you need more than that. On the other hand "system design" interviews are...strange? You, someone who never built something at the scale of these giant tech companies, are being asked to come up on the spot with a design for a system that'd scale to billions of users. There's a 100% chance that if you were to attempt building such systems and you'd never done it before, you'd discover holes in your original design left and right. Leetcode tests your logic, how you think, your IQ maybe. But system design interviews consists in regurgitating knowledge that you've crammed in your head by studying books/articles/videos on system design. It's closer to reciting poetry than problem solving. In my experience it could be replaced with multiple choice questions and you'd get the same result.

What am I missing?

[mgraczyk]

I ask a lot of system design interviews based on Facebook's products. What I look for is the ability to propose something simple, proposing the right metrics and data collection to understand scaling needs, then making reasonable guesses about which parts of the system need to scale. PHP + MySQL + Memcache is great until you also need to do ML inference (high CPU/GPU load), need to store user-uploaded video, or want to stream new content to users in near realtime (live comments).

They key is to add the minimum amount of "stuff" to a simple design to convincingly scale for some new hypothetical need.

[rr808]

Yeah I just did an interview where my design was a database, a few lambdas and a webserver and after I was thinking they must think I dont know much, I should have beefed it up a bit.

[barbazoo]

It's important to justify the design you come up with. Explain why the design is simple, pros and cons and when you'd opt for a more complicated one to solve which particular issue.

[quirino]

I imagine this is one of the posts the comment is talking about https://rachelbythebay.com/w/2020/05/07/serv/