[feld]

The problem is that Redhat and others are refusing to challenge the norm and break away from the "freeze the release; backport fixes" mantra.

Stop backporting fixes. You're forking the codebase.

Ship exactly what upstream provides.

Teach upstream projects how to do better release engineering if they're abandoning major releases to early or breaking API/ABI in a minor release.

Stop backporting fixes. You're forking the codebase.

edit: also stop incorrectly backporting security fixes and creating new CVEs. Seriously. Stop it.

[admax88q]

I think you're underestimating the stability that such practices provide for enterprise. This is what people pay Redhat for.

Not all upstreams are interested in doing release engineering. There are non zero costs to doing it. It can eat up time that can be spent on bug fixes and features, or even make it too costly to change direction if a certain approach to implementation is proving more difficult than it should be.

Look at the Linux kernel. The only reason there is a stable kernel series is because Greg K-H decided it was important enough. He was unable to convince any other developers to go along with it, and eventually the decision was "if you want to support it, then you can do it."

Do you consider the stable kernel series a fork of the codebase? Should everyone be running the newest kernel every release despite the plenty of regressions that appear?

Kernel developers are not interested in making every change in such a slow and controlled manner as to avoid any regressions. And it works for them. They get a lot of stuff done, and come back and fix the regressions later.

[tytso]

There are real tradeoffs between development velocity, stability, scope (wide/narrow applicability), and headcount.

If you don't care that much about development velocity, it's really easy to make something that is super stable.

If you only care about making things work on a very narrow use cases (to support the back end of a particular company's web servers, or just to support a single embedded device), life also gets much easier.

If you want to "move fast and break things", that's also viable.

Finally, if you have unlimited amounts of head count, life also becomes simpler.

Different parts of the Linux ecosystem have different weights on all of these issues. Some environments care about stability, but they really don't care about advanced features, at least if stability/security might be threatened. Others are interested in adding new features into the kernel because that's how they add differentiators against their competitors. Still others care about making a kernel that only works on a particular ARM SOC, and to hell if the kernel even builds for any other architecture. And Red Hat does not have infinite amounts of cash, so they have to prioritize what they support.

So a statement such as "Teach upstream projects how to do better release engineering", is positively Trumpian in its naivete. Who do you think is going to staff all of this release engineering effort? Who is going to pay for it? Upstream projects consists of some number of hobbists, and some number of engineers from companies that have their own agendas. Some of those engineers might only care about making things better for Qualcomm SOC's, and to hell with everyone else. Others might primarily interested in how Linux works on IBM Mainframes. If there are no tradeoffs, then people might mind work that doesn't hurt their interests, but helps someone else. They might even contribute a bit to helping others, in the hopes that they will help their use case. That's the whole basis of the open source methology.

But at the same time you can't assume that someone will spend vast amounts of release engineering effort if it doesn't benefit them or their company. Things just don't work that way. And an API/ABI that must be stable might get in the way of adding some new feature which is critically important to some startup which is funding another kernel engineer.

There is a reason why the Linux ecosystem is the way it is. Saying "stop it" is about as intelligent as saying that someone who is working two 25 hour part-time jobs should be given a "choice" about her healthcare plan, when none of the "choices" are affordable.

[cyphar]

I get the feeling you've never had to provide support for a distribution before. There are many guarantees that Red Hat or SUSE provide that are not provided by upstream projects. Freezing the release is the only sane way of doing it, and backporting fixes is necessary. There are exceptions to this, such as stable kernels (which was started by GregKH out of frustration of the backporting problem while at SUSE).

Upstreams don't have the resources to do proper release engineering, they're busy working on new features. The fact that SUSE and Red Hat spawned from a requirement for release engineering that upstreams were not able to provide should show that it takes a lot more work than you might think.

Also, can we please all agree as a community that writing patches and forking of codebases is literally the whole point of free software? If nobody should ever fork a codebase then why do we even have freedom #1 and #2? The trend of free software projects to have an anti-backport stance is getting ridiculous. If you don't want us to backport stuff, stop forcing us to do your release engineering for you.

[digi_owl]

Sadly more and more upstream wants to have their cake and eat it to. Just look at Flatpak, that is all about moving the updating and distribution from distros to upstream.

[cyphar]

I think Flatpak won't end up solving the problem though. Mainly because it still requires distributions to exist and provide system updates, but also because it just makes the static binary problem (that distributions were made to fix) even worse.

Honestly what I think we need is to have containers that actually overlay on the host system and only include whatever specialised stuff they need on top of the host. So updates to the host do propagate into containers -- and for bonus points the container metadata can still be understood by the host.

[digi_owl]

In the end i don't see it as a technical problem, but a mentality problem.

Again and again we see that without any financial incentive, developers are loath to put any effort into backwards compatibility and interface stability.

At the same time they all want people to be running their latest and shiniest.

So in the end, what will happen is that each "app" will bundle the world, or at least as much as they feel they need to.

[SwellJoe]

I don't get why this would be a positive thing for Red Hat's customers (or Red Hat, since stability/predictability is what Red Hat customers are paying for). There is a Red Hat-maintained Linux that is very close to upstream (Fedora). But the people who pay for RHEL don't want upstream and surprises, they want predictable for seven years and they're willing to pay a lot of money for that. Why would that be a negative for you or me? RHEL isn't breaking upstream with this practice, even if they are making mistakes in their own backports.

"edit: also stop incorrectly backporting security fixes and creating new CVEs. Seriously. Stop it."

Can you give some examples of cases where Red Hat introduced bugs in their backported patches? I follow RHEL CVEs relatively closely (because some of my packages are derived from their packages), and I can't think of an example of that happening. Debian has done so, but very rarely, that I can recall. (And, Ubuntu, too, since they just copy Debian for huge swaths of the OS.)

[noselasd]

I for one am very grateful Red Hat does not do that. We have a kernel driver for custom hardware, there's around 80 of these devices in the world, split roughly 50/50 between Windows and Red Hat users. While these devices are not cheap, we could not recoup the cost of maintaining it if we had to track the upstream kernel all the time - we tried, and could not justify the cost.

The number of times the APIs changes from under your feet is astounding - even with just keeping up with Red Hat, we spend around 4-6 times the engineering time on the driver compared to what we do with the Windows version of the driver, tracking upstream gave us almost an order of magnitude more work (And keep in mind that /only/ supporting the most recent upstream kernel is rarely an option - several versions need to be supported concurrently)

[bonzini]

Red Hat provides a stable ABI for a pretty large set of symbols. Unless you are doing strange things in the driver, a module built for RHEL 7.0 should be fine until 8 comes out.

[noselasd]

That is exactly what I am saying - which is in stark contrast to what would happen if Red Hat did not provide that stable ABI, but instead "Ship exactly what upstream provides" as the original comment suggest they should do.

[ajross]

If upstream releases were doing better release engineering in the way you mean then there would be no money to be made shipping RHEL as a product.

[krylon]

> the "freeze the release; backport fixes" mantra.

For many customers of Red Hat, that mantra is the very reason they use RHEL in the first place.

[digi_owl]

Indeed. Or they would have stuck to using Windows, or some commercial Unix.

Sadly i feel that more and more upstream wants it both ways, be able to push their latest and shiniest, and keep ignoring any need for interface stability etc.

Frankly i suspect the end result of the likes of Flatpak will be that upstream push whole distros worth of bundled libs, just so they don't have to consider interface stability as they pound out their shinies in their best "move fast and break things" manner.

[thehardsphere]

Taking this to its most ludicrous extreme, everyone should use Arch, and anyone who can't should... what? Not use Linux?

[lloeki]

The Fedora Project focuses, as much as possible, on not deviating from upstream in the software it includes in the repository.

[0]: https://fedoraproject.org/wiki/Staying_close_to_upstream_pro...

[thehardsphere]

Right, "as much as possible" implying that there are cases where this is not possible. Which is more upstream-compliant than RHEL, but not 100% "stop doing this", which is the opinion of the comment I was replying to.

[chasil]

We use both RHEL and Oracle Linux as a peer to VAX VMS and Unisys(Univac) OS2200 (a COBOL mainframe).

From the perspective of legacy systems, Red Hat's approach is more comfortable.

[gkya]

You cant teach anything to anybody. An unrelated proof: I'm having to write a bot with phantomjs to scrape my uni's announcements page and turn it into an RSS feed so that I won't have to periodically control it; because they decided wordpress wouldnt cut it and they needed some blumming angular.js stuff, breaking all the urls and removing any sort of rss feeds on the way. And all it is is a blog, basically, nothing more. Mailed them telling that I used to use that stuff, no replies in weeks. At least I'm learning phantomjs which seems to be very useful of a tool.