[fragmede]

There's a bit of magic to it, turning the OS to work well in conjunction with the hardware you have, but it's possible.

* Use 15k RPM SAS 12Gb/s drives instead of cheap consumer drives,

* Use 50 in a chassis of them instead of 12.

* Use RAID 0 instead of RAID 5 or 6 - I'll be the quoted storage space is raw, not post-raid.

* Have multiple copies of a show on disk, as the article states happens.

* Optimize for a read-heavy workflow during peak hours, (eg, mount noatime).

The devil is in the details, but what I listed above is probably a good starting point. The article states a data rate of 3GB/hour, which is only .83 megabytes/second, * 100 streams is only 83 megabytes/second, which is easy for the above configuration. Hell, I'll bet the above configuration could do 10,000 streams if the data rate is 3GB/hour with no issues given Netflix's peak read-only workload.

[thrownaway2424]

A 15000RPM SAS drive would be an _extremely_ poor choice for this type of workload. Power hungry, hot, extremely sensitive to vibration, very low data density, and, of course, ridiculously expensive.

[fragmede]

What? While I can't argue with them being ridiculously expensive (because they very much are) why would them being power hungry, hot, and extremely sensitive to vibration even matter if they are going to be rack-mounted into an ISP's data center, with redundant power-supplies and a data-center grade cooling system to match?

The only possible issue of the ones you raised is low data density, but all engineering is a trade-off, 15k rpm drives get you better seek times, which would generally lead to the ability to support more viewers in a single box - not working for Netflix, I don't know how many users they want to support per-box.

It's totally possible that between the number of streams they want to support, and the total storage in the box that they've gone with SATA drives, but to pretend that 15k RPM SAS drives are "an _extremely_ poor choice" for an enterprise-grade storage system would be to ignore the fact that Netflix is making an enterprise-grade storage appliance - and that on the top-end, those appliances commonly use SAS drives.

[thrownaway2424]

I will double-down to claim that 15krpm SAS drives are a bad choice for any application, and they are only used as a bandaid for marginal improvements on irredeemable system designs.

To address your points individually:

1) Power hungry. When you add in conversion and distribution and cooling, every watt consumed by the computer is consumed again by the datacenter infrastructure. Power costs money.

2) Hot is just the corollary of 1). Hot is also the enemy of density, and this box is very dense.

3) Sensitive to vibration. If you aren't intimately familiar with this fact then you aren't getting the performance you paid for from your 15k disks. To achieve their spectacular claimed seek times then need very careful mechanical design of their enclosure. Much more careful than racking up one of Netflix's boxes in a rack with other random vibrators.

4) Density. To get the space Netflix is using here, you need 5x more expensive 15k disks because they top out at 600GB and the ones people actually use for these workloads have 3TB.

A smart read-ahead strategy obviates the need for shiny seek time specs. For any given stream you could read ahead by 32MB or whatever. Now you've made seek time irrelevant. Put lots of RAM in the machine and you're done at a tiny fraction of the capital and operating costs of 15k disks.

[donavanm]

+1, in general. To adda bit of content:

1) Power Hungry, the total power envelope of a rack position is the limiting issue. You can fit 16-20 disks per RU no problem. The constraint on total density is your 5kVa or 10kVa power budget. Watts matter.

3) sensitive to vibration, be very very quiet http://www.youtube.com/watch?v=tDacjrSCeq4

4) Density, actually 4TB is hitting the sweet spot for $ per byte last time I looked. If you need absolute density look for 5 & 6TB Real Soon Now. If you can tolerate some loss variable density disks around 5TB look quite a bit more cost effective.

5) Read ahead, you only need to read ahead by a couple of chunks. For video ball park it at 2MB. You don't really need lots of RAM, think 32-64GB per chassis. Additionally each 8GB dimm costs about the same power as a disk. By going with 32GB instead of 64 I can fit another 2-3 disks in the power budget per chassis.

[cjg_]

They are actually using standard SATA drives, see spec here, https://www.netflix.com/openconnect/hardware

[fragmede]

"The following system was developed and first deployed at the end of 2011."

[cjg_]

Newer revisions use 4 or 6 TB drives and such 15K SAS drives does not exist from what I can find.

See http://oc.nflxvideo.net/docs/OpenConnect-Deployment-Guide.pd...