[CPlatypus]

He only gets it half right. A filesystem interface instead of a block interface is the right choice IMO. Private storage instead of distributed storage is the wrong choice for capacity, performance, and (most importantly) availability reasons. They didn't go with a ZFS-based solution because it was the best fit to requirements. They went with it because they had ZFS experts and advocates on staff.

As Schopenhauer said, every man mistakes the limits of his own vision for the limits of the world, and these are people who've failed to Get It when it comes to distributed storage ever since they tried and failed to make ZFS distributed (leading to the enlistment of the Lustre crew who have also largely failed at the same task). If they can't solve a problem they're arrogant enough to believe nobody can, so they position DAS and SAN as the only possible alternatives.

Disclaimers: I'm the project lead for CloudFS, which is IMO exactly the kind of distributed storage people should be using for this sort of thing. I've also had some fairly public disputes with Bryan "Jackass" Cantrill, formerly of Sun and now of Joyent, about ZFS FUD.

[ssmoot]

ZFS is just the FS. But you know that already.

The SAN solutions they migrated to are not ZFS based. Unless I'm mis-remembering (I read this a couple days ago) they were only using ZFS to slice LUNs.

Point is, you're taking pot-shots at ZFS when the main thrust appears to be: "It was hard to make iSCSI reliable. Once we did, by buying expensive storage-vendor backed solutions, we found it wasn't financially compelling."

They're a hosting provider. If it takes a replicated SAN pair (which is the wrong way to go about it BTW, though admittedly it's the way the storage vendors and their "appliance" mentality would have it done) to service just a pair of VM hosts (they're still using Zones right?) then it just didn't make sense money-wise for them. If they planned capacity to provide great performance, they weren't making enough money on the services for what they were selling them for.

That's not an "iSCSI is unreliable" problem. It's not a "networked storage is broken" problem. It's not a "networked storage is slow" problem. It's not even a "ZFS didn't work out" problem.

If you go out and spend major bucks on NetApp, not only are you going to have to deal with all the black-box-appliance BS, but it's going to cost a lot of money. A LOT. And DAS is going to end up cheaper to deploy, maintain, and your margins are going to be a lot higher.

DAS is the right choice for a hosting provider who wants to maximize their profits in a competitive space.

It's not the best choice for performance, availability or flexibility for clients though. So you have to ask yourself what kind of budget you have to work with, and what goals are important to you?

BTW, there's _budget_, and then there's NetApp/EMC budget. Just because you need/want more than DAS can give you doesn't mean you need to tie your boat to an insane Enterprise grade budget.

[CPlatypus]

Perhaps you should RTFA. The author says explicitly that what they do now is "lean on ZFS" and "keep the network out of the storage solution" which made their provisioning more complex because they could no longer treat local disks as ephemeral (i.e. that data can't be assumed to exist anywhere else). I knew this when I wrote the GP. My whole point is that they treated it as a "networked storage is broken" problem even though it wasn't, because of their "ZFS is the only tech we need" bias. Thanks for re-stating that.

As for "DAS is the right choice" that's just wrong on many levels. First, people who know storage use "DAS" to both private (e.g. SATA/SAS) and shared (e.g. FC/iSCSI) storage, so please misusing the term to make a distinction between the two. Second, I don't actually recommend either. I don't recommend paying enterprise margins for anything, and I don't recommend more than a modicum of private storage for cloud applications where most data ultimately needs to be shared. What I do recommend is distributed storage based on commodity hardware and open-source software. There are plenty of options to choose from, some with all of the scalability and redundancy you could get from their enterprise cousins. Just because some people had some bad experience with iSCSI or DRBD doesn't mean all cost-effective distributed storage solutions are bad and one must submit to the false choice of enterprise NAS vs. (either flavor of) DAS.

In short, open your eyes and read what people wrote instead of assuming this is the NAS vs. DAS fight you're used to.

[ssmoot]

They "lean on ZFS" for DAS.

Seriously. You tell me. What does that have to do with your rant on ZFS? It could have as well been an LSI controller doing RAID6. Or mdadm. Doesn't matter.

That's the evolved solution they came up with.

The "networked storage is broken" pitch actually comes in with the EMC/NetApp interim solution as well. I don't buy it either, but it's a joke to claim the problem was ZFS on the Zones when the Targets weren't running ZFS.

You're awfully prickly, but I didn't suggest it came down to "Enterprise" NAS vs DAS. I actually think networked storage is here to stay (and that's a good thing).

I have my doubts we'll see a stable, inexpensive (or free) Distributed or Clustered file-system ready to replace traditional solutions anytime soon. I'm happy to see people try though.

You clearly have an axe to grind with ZFS though. In my experience it's been by far more stable than any available Linux FS I've used. Pull the power again and again, replace and resilver all you want. Manage terabytes and don't worry about corruption. I wouldn't trust ext3/4fs for anything I couldn't stand to lose...

PS: http://en.wikipedia.org/wiki/Direct-attached_storage

"People who know storage". I don't see iSCSI on that list. Nor FCoE. DAS (at least according to Wikipedia) explicitly rules out switching. Which is how I've always viewed it.

[CPlatypus]

You're really not getting it, are you? I never said ZFS was the problem, as you seem to think. I'm just saying it's not the solution either. It's a crappy solution, failing to protect against host failures and creating myriad problems in provisioning around the fact that each VM's storage is stranded on one node until it's explicitly copied somewhere else. And if you don't think there are decent distributed filesystems out there, you're just not keeping up with the field and shouldn't be commenting on it.

[ssmoot]

I don't think I am getting it no. You don't think ZFS is the problem?

So you aren't calling ZFS a "crappy solution"? Just the DAS usage?

What is your gripe exactly then? The overblown critique of networked storage? Well we agree on that at least then. I think.

Honestly, with all the "read the fucking article", it's-not-DAS, oh-it-is, CloudFS is way moar better than ZFS, I never said ZFS sucked, "Bryan ZFS Cantrill is a jackass" you've left me absolutely bewildered at what your intended point (if any) actually is?

For the record, my only comment on (free) distributed filesystems (that aren't vendor-locked and actually unusable to me) is that I wouldn't personally trust them with my data. Not until they have the features I need, and then are running out in the wild, widley deployed for a couple years so I'm not a guinea pig.

I'll even throw you a bone: Even just last year ZFS was having major melt-downs when a new inadequately vetted feature was added. A few years ago it wasn't uncommon to face corruption when trying to do fairly routine things managing disks. Bugs can and do happen.

Maybe CloudFS, or Gluster is ready for prime-time, housing terabytes of data reliably and never making a misstep. I just don't think it's smart to bet your business on it. Not at least without a plan B since moving data around isn't an option when you're down and have terabytes you need to get back online.

[dasil003]

I'm wait out of my depth on this stuff, but the arrogant tone of the OP really rubs me the wrong way given my personal experiences with Joyent in the past. They act like their shit doesn't stink and they are god's gift to tech, when in reality they just like playing with cool toys and they're not providing better service than anyone else.

[Nrsolis]

Do you have any opinions of Ceph? They seem to be dong a similar thing.