Hacker News new | ask | show | jobs
by handruin 3426 days ago
I've gone all-in on HGST for personal use because I've seen their results from Backblaze and also have good luck with them so far. I have 14 of their 4TB NAS and 22 of the 6TB NAS.
4 comments

I'm amazed that personal use requires 188TB of storage.

That's more than my employer's Hadoop cluster...

I could see that getting filled up easily with a vast 1080p TV/Movie collection. Movies coming in at 1080p eats up disk really quickly.

Uncompressed 4k @ 24fps, at 10bit color comes in at 324MB/s, so just being a wedding videographer could be 1.1T per hour of video. Any given small project could eat up 15-20T per project.

I never said I required this much for personal use...I just have it. :)

I started with 12 x 4TB in one NAS and I filled it in a little over 2 years. I just recently built a second NAS with 20 x 6TB to last me hopefully for the next several years.

The other remaining drives unaccounted for are in two workstations for local storage.

Maybe he does his own offsite backups with "personal use" HD's in several physical locations, each in a highly-redundant raid configuration?

Even so it seems like a lot.

Someone's gotta back up that climate data...
Actually...I am doing some of that also.
Small hadron collider?
Perhaps a small hardon collider.

(Context: http://archive.fortune.com/2006/11/30/magazines/fortune/obri... )

I was debating whether I should "favorite" this thread. You made my decision for me, thank you.
Be warned, I got an HGST drive for personal use, happy to pay the premium. It was quite loud, my wife commented from across the room "What is that?". A periodic loud "tick". Turns out it does some kind of calibration and you can't turn it off. Many reviews mentioned it, didn't even look.

I returned it.

WD blacks do that too, although it's more of a grunty, almost snort-like oink.
> 188 TB of storage

> personal use

Who needs that much storage for personal use?

Anyone working with video, 3D animation, storing RAW photography (and taking 100's of photos on a daily basis), and I can imagine a few other purposes. Especially if you want an on-site backup of that data.

I've bought 10TB of storage space every year for the past 3 years and don't see myself stopping. I need the storage space! If these data-heavy hobbies were instead my daytime job I could easily see myself needing 100TB+ in storage (and then 100TB+ of backups). I fill roughly 8TB/year in storage space (4TB of data + 4TB backup data).

Some people believe in redundant backups. I only have redundant backups of extremely important information - which are mostly text documents so don't take up much space. But if someone was storing 3 copies of all of their media assets you end up using a lot of space. For example, 10TB of video turns into 30TB of video. And 180TB is really only 60TB of data, which isn't that much for data heavy hobbies.

Mostly Linux ISOs.
You wouldn't happen to frequent a certain, data focused, subreddit now would you?
All of them?
Someone shooting a lot of 4K or even HD video, for example.
From a little googling, 4k video appears to take about 22GB an hour[1], so 188TB would beover 8,500 hours of 4k video[2]. That's almost an entire year of constant 4k video. That's a lot of storage for personal use.

1: Not sure what encodig

2: Napkin calculation, corrections welcome

Not just encoding, but it also depends on a number of things like color depth, chroma subsampling (4:2:2), and so on. It's not uncommon for a single film to approach half a petabyte of raw in a digital world with a "typical" setup (4:2:2, ProRes maybe, 10-bit, etc). To use an example, I think I read somewhere that Gone Girl shot a few hundred TB for a technically straightforward film. Some cameras, particularly when you start getting to digital cinema (which is > "4K"), can shoot as much as a terabyte and a half per hour. There are 8K sensors in common availability now; RED has one that can shoot 75fps 8K at 2.4:1, which I don't even want to calculate, but that I know for a fact it can't write to its own storage at native bitrate. Productions eat storage these days and moving that data around is a challenge.

Hint, hint, to clever founders: hard drives in Hollywood.

22 GB for an hour of 4k video is fairly compressed. Raw 4k footage runs closer to 300 GB hour.
Don't forget parity drives for availability take away from usable storage.

NAS #1 is 12 x 4TB with 2 drives used for parity.

NAS #2 is 20 x 6TB with 4 drives used for parity.

That takes away roughly 32TB of usable space under zfs. NAS #1 has 40TB usable. NAS #2 has 96TB usable. Total 136TB usable.

The remainder of the drives not accounted in the list above are in workstations for local storage and other random tasks.

> From a little googling, 4k video appears to take about 22GB an hour

That's "home delivery" video, not the stuff you work on which would be far less compressed or even uncompressed, and would include reams of data thrown out entirely at the final production stages.

Think about the difference between "work" (mixing/mastering/production) audio data (uncompressed and at 24 or 32 bits) versus "consumer" audio (mp3/aac at 48/16).

Editing also tends to create a lot of huge temporary and intermediate files that will eat up disk space in a hurry. 188TB is a lot of space, but nowhere near unreasonable for someone doing video work. Plus, as other people have mentioned you need to account for filesystem overhead, redundancy (RAID), backups, TB vs. TiB, and so on.
My 1440p 4:4:4 videogame captures at 60 fps come out to about 700 GB per hour, sometimes more depending on the scene complexity. That would fill up a 200 TB array in no time.

I'd record to ProRes or DNxHR if I could, but there's no viable method for doing that on Windows without using an FFMPEG pipeline (which won't do 4:4:4 color sampling with those codecs and simply uses too much CPU at 4:2:2).

Why do you do that?
I would haircut the number for RAID then probably half it if he uses a backup.
I've done the same, in smaller quantity (12x 2TB drives.) I wouldn't buy any other brand, after having been burned by Seagate and WD too many times. These all run 24x7 on my home network.