[avar]

> Pretty colors when running dmesg!

Neat, but a great isolated example of the ancient software people who use RHEL have to deal with. RHEL 7 has dmesg from util-linux 2.23, the "colors by default" feature[1] first came out with 2.24 released on October 21st, 2013, which is around the time[3] the first beta of RHEL 7 came out.

1. https://github.com/karelzak/util-linux/commit/9bc2b51a06dc9c...

2. https://github.com/karelzak/util-linux/releases/tag/v2.24

3. https://en.wikipedia.org/wiki/Red_Hat_Enterprise_Linux#RHEL_...

[lima]

Most things on RHEL 7 aren't ancient - there's plenty of backports, even major ones.

Some examples:

- OpenSSL rebase to 1.0.2k (for HTTP/2 support).

- overlayfs2 kernel support.

- Kernel eBPF instrumentation.

- Introduction of podman and friends.

- Ansible is kept up-to-date.

- GCC 7 and Python 3.6 via Software Collections.

This includes extensive testing. I have non-production systems on Fedora which run mainline kernels and have seen my fair share of performance regressions and crashes.

I'm assuming there was no notable customer demand for colorful dmesg output.

[X-Istence]

Some of the backports aren't nearly as fast or performant as using a newer kernel though.

For example eBPF was back ported (and also in CentOS), but running a syscall heavy work-load in a docker container on the older kernel about 50% of the CPU time was spent in the kernel filter.

I ended up moving our entire CI/CD platform to Ubuntu 18.04 and the performance issues went away and my workloads now run at full speed without slowdowns.

RHEL 8 comes with the 4.18 series of the Linux kernel that is already EOL. That's a shame and once again it will fall behind quickly :/

[CameronNemo]

Seriously why did they not up it to 4.19? Do they hate bicycles? Do they make money based on the fact that upstream LTS kernels have short shelf lives compared to RHEL's own LTS kernels?

[rwmj]

The kernel version was finalized some time before this release when 4.18 was current. Red Hat expends a ton of effort on long term maintenance and huge backports of new features to the kernel, so while I don't want to speak for the kernel team, I don't think the upstream stable kernels bring very much to the table.

Plus (my personal view) what goes into the upstream stable kernel is fairly random based on just mailing list NACKs, whereas what goes into the RH kernel has to pass a massive range of automated tests on a wide variety of real hardware.

[snuxoll]

Red Hat also certifies a whitelist of symbols that will remain identical in all future releases for the life cycle, stuff like that takes time so just grabbing the newest LTS kernel the moment they cut a release isn’t feasible.

[paulcarroty]

> That's a shame and once again it will fall behind quickly

RH use kernel with TONS of patches, so, version isn't critical here.

[pas]

Their frankenkernel is very hit and miss, backports or not. Docker had to disable a few features, because at first they seemed to work, but then they were buggy on RHEL.

[Spivak]

But why wouldn't you use Red Hat's build of Docker? I mean if you're already paying them...

[paulcarroty]

Can't remember any real problems with docker on RHEL/Fedora.

[X-Istence]

I mentioned my sys call filtering issue with eBPF taking up massive amounts of CPU time...

[VintageCool]

Colorful dmesg output was already available on Cent7, just not the default.

[CoolGuySteve]

I've always felt that RHEL really excels at that old school corporate Unix feel of having to deal with stodgy tools that are either really old and/or lack basic ease of use features.

Reminds me of the time I wrote a script that called 'hostname -x' on SunOS instead of Solaris and it changed the hostname to '-x' and broke X11. RHEL is the nostalgia Linux.

But seriously, has anyone ever empirically verified that the Debian Stable/RHEL model of shipping a bunch of really old packages and then layering years of patches over top actually generates more stable, more secure code?

My intuition after a couple decades of software dev is that bugs will fester longer in the old version and the patches themselves will start having bugs as the top of tree diverges more and more from the shipped package over time.

[avar]

The main source of stability for RHEL isn't that any one arbitrary version of a package they ship is better than another one, or that their patches on top don't suck. It's that they ship a long-term "stable" (as in "doesn't change much", not "sucks less") set of software for production use.

Thus, if you install some random vendor's shitty software you can rest assured that the version of libcurl and 50 other libraries they depend on is something they themselves have tested on RHEL.

The same goes for hardware that you buy. When you buy e.g. Dell rack-mounted servers you can safely assume that the open source driver version maintained by the vendor shipped as part of the RHEL kernel is something that's seen extensive production use, unlike the latest upstream kernel, or whatever "in-between" Debian et al are shipping.

Am I recommending you use RHEL? No, it's not the right answer for everything, and I certainly have my share of RHEL scars, including a couple of times where a mundane bug in my program turned out to be a kernel bug (one in RHEL's own shitty patches, another "known" bug with their ancient kernel).

But this is the reason to use it, and why some major commercial vendors say "we support Linux, as any distro you want as long as it's on this list of RHEL versions". They just want to deal with those kernel/library versions, not any arbitrary combination out there in the wild.

[_red]

>layering years of patches over top actually generates more stable, more secure code

Well, I think your definition of 'stable' is different than what RHEL/Debian customers think. Stable isn't seen as "doesn't have bugs", its "works predictably". Which is a subtle but meaningful difference.

[secabeen]

> But seriously, has anyone ever empirically verified that the Debian Stable/RHEL model of shipping a bunch of really old packages and then layering years of patches over top actually generates more stable, more secure code?

Debian has released a new stable version every 2 years for the last 14 years. RHEL/CentOS are the only ones on a 3-5 year cycle.

[CameronNemo]

And yet they wait months between freezing the distribution and releasing, because a few troublesome packages have issues.

Someone needs to thaw Debian out.

[rincebrain]

No?

The fact that there's a freeze to allow for shaking out troublesome issues in a few packages (and possibly discover ones you didn't already find in older ones) without much risk of others newly breaking is a feature, not a bug.

Debian testing/unstable, backports and third-party repos exist if people really want the latest anyone's packaged, or the latest version of one specific thing on their otherwise stable system.

You may disagree with the philosophy, but every part of that behavior is working as intended.

[linuxftw]

Stable doesn't necessarily mean 'doesn't crash' what it means is that the API/ABI interface is stable.

EG: let's say libfoo.so.1 implements DO_FOO; libfoo.so.2 implements DO_FOO2, but not DO_FOO. In this case, anything you need that links to libfoo.so.1 and needs DO_FOO would need to be patched, recompiled, and shipped out to all your customers. For the distribution provider, this is not really a huge deal. But RHEL is merely the platform. The value-add is that 3rd parties can write software and compile against libs and know they're not going to break arbitrarily.

Similarly, if you've ever written a kernel driver, you'd know that kernel function names and signatures can change from release to release. The same example above applies to kernel code as well. So, compiled binary drivers would have to be patched and recompiled, and shipped out. If you're writing a driver for a network card, would you prefer having to ship your (non bug) driver updates every few months, or every few years?

[geofft]

It doesn't generate more secure code—as you say, patches themselves may have bugs, and way fewer people are looking at the patched branches. Active development happens on HEAD, and dodgy code is often rewritten before anyone goes actually looking for bugs (security or otherwise). Many years ago I helped with a paper on how the practice of applying only "important" security bug fixes doesn't work: https://arxiv.org/abs/0904.4058

But the goal of the long-term-stable approach isn't security or stability per se: it's striking a tradeoff between operator work and risks to security and stability. You could, of course, snapshot Fedora (or Debian testing, or Arch, or whatever) from 2013 and keep running it. Nobody is stopping you, and it'll still run on new machines. And then you have to do zero work to keep your system up-to-date, but you'll likely have tons of security and stability bugs. On the other extreme, you could run Fedora rawhide (or Debian unstable, or current Arch, or whatever) and update nightly, which would mean you get security fixes as fast as possible (they're almost always developed on HEAD and backported to release branches), and you get performance and stability fixes that people haven't deemed worth backporting, but you also risk API-incompatible changes that break the actual applications you care about. You'll need to set up really good CI to make sure you have coverage of everything in your application, and it's not just a matter of automation: you'll need a well-staffed team to respond quickly every time that CI goes red, figure out what changed, and update your applications to match. (And, of course, you have the risk of security issues in new code that hasn't been subject to public scrutiny yet—the inverse problem of security issues in old code that's no longer subject to public scrutiny.)

The goal of a long-term stable distro is to be in the middle of those two, to give you something that changes rarely (stability in the sense of "no surprises," not "doesn't crash in prod") but often enough that you get major, identified security fixes and particularly safe performance (and stability-as-in-"no longer crashes in prod") fixes.

And yes, part of the goal of a long-term stable distro is that it provides you measurable security and stability over unmeasurable but potentially greater security and stability. They don't fix every CVE, but they do fix the flashy ones. You can look at it cynically and say, this is the distro for people who want to tell their boss "Yes, we patched Heartbleed and Shellshock" but don't inherently care about security. But on the other hand, flashy vulnerabilities are more likely to be exploited, so it's not a particularly bad tradeoff.

[mistrial9]

.. booting a machine with Ubuntu 1404 as this is written, it is NOT so easy as 'snapshot and run it forever' because the OS people are trying to HELP you by FORCE to get a current version, plus so much of these machines success was networking, is network based, and relies on network to operate more things that anyone casually realizes..

It is a GOOD thing to run old versions, for purpose, by your personal choice. It is NOT GOOD to have help by force, and in the US law system at least, many individual rights are based on this assumption, even with some inevitable negative outcomes. Please note that in many parts of the world, and in many kinds of organization, this trade-off is NOT made, and quite a few fundamental technical decisions are going to be made along the lines of 'do it, there is no choice'

[lima]

As you say, it goes both ways. Many kernel vulnerabilities are found and fixed within weeks to months of introducing them, with LTS distros totally unaffected.

And then you have bugs being fixed on master (sometimes silently), and the backport maintainers fail to backport them.