Hacker News new | ask | show | jobs
by bpodgursky 1850 days ago
It's interesting, because I think HDFS (and NameNodes in particular) were impressively engineered for a use-case which didn't quite materialize — ie, very fast metadata queries (they are still much faster than S3 API calls). Turns out that cheap, simple, and massively scalable object storage is just far far more important in practice.

I think there are still a couple use-cases where HDFS dominates S3 (I think some HBase workloads?). But yeah, I scaled up and maintained a 2000+ Hadoop cluster for years, and I would never choose it over object storage if given any plausible alternative.
2 comments
macksd 1850 days ago
This is actually a topic I love to talk about because I spent a lot of my time on S3A and the cloud FileSystem implementations. Fast metadata queries were actually a huge deal for query planning, and of course with performance there were a lot of potential surprises on S3. HBase was (unsurprisingly) heavily dependent on semantics that HDFS has but that are hard to get right on object storage, and required a couple of layers to be able to work properly on S3 (and even then - write-ahead logs were still on a small HDFS cluster last I heard). My biggest complaint about S3 was always eventual consistency (for which Hadoop developed a work-around - it originally employed a lot of worst-practices on S3 and suffered from eventual consistent A LOT) but now that S3 has much better consistency guarantees, I agree: it's incredibly hard to beat something that cheap.
link
fendale 1849 days ago
For a job that needs to access 100's of thousands of small files, the ability to read the meta data quickly is very important.

This is the wider issue with small files. On HDFS each file uses up some namenode memory, but if there are jobs that need to touch 100k+ files (which I have seen plenty of), that puts a real strain on the Namenode too.

I have no experience with S3 to know how it would behave in terms of metadata queries for lots of small objects.

link
dikei 1849 days ago
Small files with S3 is both slow and expensive too. But at least one bad query won't be able to kill your whole cluster like HDFS.
link