Hacker News new | ask | show | jobs
by pgaddict 3241 days ago
I think a natural follow-up question is "Why Red Hat does not have engineers to support btrfs?" That is, if the lack of engineers is a symptom, what is the cause?

I'm pretty sure, had RH wanted they could either hire or assign engineers to maintain the btrfs code, take care of patches from upstream, etc. So why didn't that happen? I wonder what is your opinion on that.

I see a bunch of possibilities (not necessarily independent ones):

1) Politics. Perhaps RH wants to kill btrfs for some reason?

I see this as rather unlikely, as RH does not have a competing solution (unlike in the Jigsaw controversy, where they have incentives to kill it in favor of the JBoss module system).

2) Inability to hire enough engineers familiar with btrfs, or assign existing engineers.

Perhaps the number of engineers would be too high, increasing costs. Especially if not only to maintain the RHEL kernels, but to contribute to btrfs and move it forward.

Or maybe there's a pushback from the current filesystems team, where most people are xfs developers?

3) Incompatible development models.

If each release requires a rebase, perhaps supporting btrfs would require too much work / too many engineers, increasing costs? I wonder what Suse and others are doing differently, except for having in-house btrfs developers.

4) Lack of trust btrfs will get mature enough for RHEL soon.

It may work for certain deployments, but for RHEL customers that may not be sufficient. That probably requires a filesystem performing well for a wider range of workloads.

5) Lack of interest from paying RHEL customers.

Many of our customers have RHEL systems (or CentOS / Scientific Linux), and I don't remember a single one of them using btrfs or planning to do so. We only deal with database servers, which is a very narrow segment of the market, and fairly conservative one when it comes to filesystems.

But overall, if customers are not interested in a feature, it's merely a pragmatic business decision not to spend money on it.

6) Better alternatives available.

I'm not aware of one, although "ZFS on Linux" is getting much better.

So I tend to see this as a pragmatic business decision, based on customer interest in btrfs on RHEL vs. costs of supporting it.

2 comments

All this talk about Oracle is just plain stupid. Oracle doesn't control anything, the community does. One core developer still works on Btrfs from Oracle, the vast majority of the contributions come from outside Oracle.

Now as to

> "Why Red Hat does not have engineers to support btrfs?"

You have to understand how most kernel teams work across all companies. Kernel engineers work on what they want to work on, and companies hire the people working on the thing the company cares about to make sure they get their changes in.

This means that the engineers have 95% of the power. Sure you can tell your kernel developer to go work on something else, but if they don't want to do that they'll just go to a different company that will let them work on what they care about.

This gives Red Hat 2 options. One is they hire existing Btrfs developers to come help do the work. That's unlikely to happen unless they get one of the new contributors, as all of the seasoned developers are not likely to move. The second is to develop the talent in-house. But again we're back at that "it's hard to tell kernel engineers what to do" problem. If nobody wants to work on it then there's not going to be anybody that will do it.

And then there's the fact that Red Hat really does rely on the community to do the bulk of the heavy lifting for a lot of areas. BPF is a great example of this, cgroups is another good example.

Btrfs isn't ready for Red Hat's customer base, nobody who works on Btrfs will deny that fact. Does it make sense for Red Hat to pay a bunch of people to make things go faster when the community is doing the work at no cost to Red Hat?

Oracle certainly controls the license for ZFS.

Release under a compatible license would likely see a ZFS kernel module appear in EPEL immediately; Red Hat would likely replace XFS with ZFS as the default in RHEL8 were this legally possible.

Oracle supports BtrFS in their Linux clone of RHEL. It certainly appears that Red Hat is swallowing a "poison pill" to increase Oracle's support costs (and I'm surprised that they have not swallowed more).

http://docs.oracle.com/cd/E52668_01/E54669/html/ol7-about-bt...

With these new added costs, Oracle might find it cheaper to simply support the code for the whole ecosystem (CentOS and Scientific Linux included). Given the adversarial relationship that has developed between the two protagonists, an enforceable legal agreement would likely be Red Hat's precondition.

Otherwise, BtrFS has been mortally wounded.

> Oracle doesn't control anything

Oracle owns a lot of patents and I suspect both ZFS and BtrFS rely on some.

> All this talk about Oracle is just plain stupid. Oracle doesn't control anything, the community does. One core developer still works on Btrfs from Oracle, the vast majority of the contributions come from outside Oracle.

FWIW I haven't said anything about Oracle & btrfs ...

>> "Why Red Hat does not have engineers to support btrfs?"

> You have to understand how most kernel teams work across all companies. Kernel engineers work on what they want to work on, and companies hire the people working on the thing the company cares about to make sure they get their changes in.

> This means that the engineers have 95% of the power. Sure you can tell your kernel developer to go work on something else, but if they don't want to do that they'll just go to a different > company that will let them work on what they care about.

> This gives Red Hat 2 options. One is they hire existing Btrfs developers to come help do the work. That's unlikely to happen unless they get one of the new contributors, as all of the seasoned developers are not likely to move. The second is to develop the talent in-house. But again we're back at that "it's hard to tell kernel engineers what to do" problem. If nobody wants to work on it then there's not going to be anybody that will do it.

Sure, I understand many developers have their favorite area of development, and move to companies that will allow them to work on it. But surely some developers are willing to switch fields and start working on new challenges, and then there are new developers, of course. So it's not like the number of btrfs developers can't grow. It may take time to build the team, but they had several years to do that. Yet it didn't happen.

> And then there's the fact that Red Hat really does rely on the community to do the bulk of the heavy lifting for a lot of areas. BPF is a great example of this, cgroups is another good example.

I tend to see deprecation as the last state before removal of a feature. If that's the case, I don't see how community doing the heavy lifting makes any difference for btrfs in RH.

Or are you suggesting they may add it back once it gets ready for them? That's possible, but the truth is if btrfs is missing in RHEL (and derived distributions), that's a lot of users.

I don't know what are the development statistics, but if majority of btrfs developers works for Facebook (for example), I suppose they are working on improving areas important for Facebook. Some of that will overlap with use cases of RHEL users, some of it will be specific. So it likely means a slower pace of improvements relevant to RHEL users.

> Btrfs isn't ready for Red Hat's customer base, nobody who works on Btrfs will deny that fact. Does it make sense for Red Hat to pay a bunch of people to make things go faster when the community is doing the work at no cost to Red Hat?

The question is, how could it get ready for Red Hat's customer base, when there are no RH engineers working on it? Also, I assume the in-house developers are not there only to work on btrfs improvements, but also to investigate issues reported by customers. That's something you can't offload to the community.

I still think RH simply made a business decision, along the lines:

1) The btrfs possibly matters to X% of our paying customers, and some of them might leave if we deprecate it, costing us $Y.

2) In-house team of btrfs developers who would work on it and provide support to customers would cost $Z.

If $Y < $Z, deprecate btrfs.

Wholeheartedly agree that btrfs isn't ready for a real customer base. I wish SuSE would have learned that lesson before they pushed it as default.
It feels like reiserfs all over again.
Well, I am guilty of using reiserfs selectively (and with research + design) and having a really good experience with it. Maybe I should have done the same with btrfs but I took btrfs on faith and was burned.
Oracle has essential control of both "nextgen" filesystems that should be used in Linux - as Sun, they developed and licensed ZFS, and they are the chief contributors of BtrFS. Their refusal to release ZFS under a license that is compatible with the GPL is keeping it out of Red Hat's distribution.

This move by Red Hat must be seen as a provocation of Oracle, to force either greater cooperation and compliance in producing a stable BtrFS for RHEL, or the release of ZFS under a compatible license. Red Hat has put an end to BtrFS for now, and Oracle will have to go to greater lengths to use it in their clone. Customers also will not want it if it does not run equally well between RHEL and Oracle Linux.

It is obvious that Oracle will have to assume higher costs and support if they want BtrFS in RHEL. Red Hat is certainly justified in bringing Oracle to heel.

Oracle recently committed preliminary dedup support for XFS, so they must be intimately aware of the technical and legal issues behind Red Hat's move.

https://blogs.oracle.com/linuxkernel/upcoming-xfs-work-in-li...

Oracle is not the "chief contributors" of Btrfs. If anyone is, it's Facebook. Chris Mason (the btrfs creator) worked for Oracle. He left in 2012.

> This move by Red Hat must be seen as a provocation of Oracle

I doubt it.

A thousand pardons - I am mistaking "initially designed" for current control.

https://en.wikipedia.org/wiki/Btrfs

"... initially designed at Oracle Corporation for use in Linux."

Oracle uses RHEL for their Unbreakable Linux [0] distribution. The least thing they can do is open up ZFS for the Linux community.

[0] - https://linux.oracle.com/

Seconded. I am in absolute agreement.
> open up ZFS for the Linux community.

More likely they'll support it only on their Linux.

This will particularly impact the "Red Hat Compatible Kernel" (RHCK) that is shipped by Oracle Linux.

https://docs.oracle.com/cd/E37670_01/E57668/html/ol_kern_65r...

Assuming that RHEL v8 strips BtrFS, Oracle's RHCK will have to add support back in, and thus no longer be "compatible." Without that support, some filesystems will fail to mount at boot. In-place upgrades from v7 to v8 will be problematic.

Oracle has worked very hard to maintain "compatibility" with Red Hat, even going so far as to accept MariaDB over MySQL. Their reaction to the latest "poison pill" will be interesting.

Why would Oracle have to add Btrfs support back into the RHCK? It's exactly the point of this kernel to be 100% identical to upstream RHEL. If an Oracle Linux user needs Btrfs support, it will still be included in the "Unbreakable Enterprise Kernel" (UEK), which Oracle provides as an alternative.
Any BtrFS filesystems in /etc/fstab won't mount if/when an RHCK boots that lacks the filesystem driver.

An in-place upgrade from v7 to v8 could easily get hosed.

Does Oracle support Btrfs (as opposed to making it just a tech preview) with the compatible kernel? I don't think so, since it's the same code as RHEL. And if not, hosing in-place upgrades is acceptable. RHEL 7 is supported until 2024.