Hacker News new | ask | show | jobs
by monstrado 4367 days ago
> The NameNode is a single point of failure.

The NameNode service supports high availability out of the box, and uses a QJM (Quorum Journal Manager) to share edits between the active/standby node. To say the NameNode is a SPOF is not only misleading, it's incorrect.

> the NameNode is also a scaling bottleneck. Services which build on Hadoop

What's indicating that the NameNode is a scaling bottleneck? There are production Hadoop clusters with PBs of data, and the NameNode has not been an issue for scalability. Just so we're clear, data written to HDFS does not flow through the NameNode.

> As a result, many enterprises that use Hadoop and HBase do so only in non-critical services, such as supporting the business analytics group

This is the point where I point to the hundreds of companies which rely on Hadoop to run mission critical applications (Facebook, Yahoo, Ebay, Box, ..). Just google around...look at customers of Cloudera, Hortonworks, Pivotal, etc.
2 comments
spullara 4367 days ago
The name node is a significant vertical scaling challenge. It is one of the reasons that Yahoo limits their cluster sizes.
link
ithkuil 4367 days ago
That's right, it should be noted however that the need of horizontally scaling the equivalent of the NameNode kicks in only when you really have a very large storage system, where large can be defined as:

"You know you have a large storage system when you get paged at 1 AM because you only have a few petabytes of storage left." [1]

Even if you are below that size you might have a large system, and even if using a single metadata master node might be a sound solution you still have lot of interesting problems to solve. Don't make systems more complex than necessary.

More pointers to google public material on colossus in [2].

[1] http://static.googleusercontent.com/media/research.google.co...

[2] http://www.highlyscalablesystems.com/3202/colossus-successor...

link
monstrado 4367 days ago
Very few cluster sizes in enterprises reach this limit, and by limit I mean 1k -> 2k nodes. In reality, there's very little demand for namenode scalability at the current moment, but once this changes, the community will implement it.
link
topher-the-geek 4366 days ago
> Just so we're clear, data written to HDFS does not flow through the NameNode.

I wrote that poorly. You're right, file blocks do not flow through the NameNode, only metadata updates do. The issues occur when the clients' metadata operations saturate the node.

> This is the point where I point to the hundreds of companies

Indeed. Many companies have had great success with Hadoop. It's one of the reasons Hadoop sees such broad use. However, other users' mileage has varied.

link