That said, if you're really in the position of depending on a free project for over five years of security support, you probably will be totally fine with just ignoring the fact it's out of support. Just keep running Debian 6 for a decade, whatever. The code still runs. Pretend you've patched. Sure, there are probably some vulnerabilities, but you haven't actually looked to see if the project you're actually using right now has patched all the known vulnerabilities, have you?
RHEL kernel versions are basically incomparable with vanilla kernel versions. They have hardware support and occasionally entire new features that have been backported from newer kernels in addition to the standard security & stability patches.
This means that RHEL 7 using a "kernel version" from 2014 will still work fine with modern hardware for which drivers didn't even exist in 2014.
That is not a good thing. RH frankenkernels can contain subtle breakage. E.g. the Go and Rust standard libraries needed to add workarounds because certain RH versions implemented copy_file_range in a manner that returns error codes inconsistent with the documented API because patches were only backported for some filesystems but not for others. These issues never occurred on mainline.
And for the same reasons that the affected users chose a "stable" and "supported" distro they were also unable to upgrade to one where the issue was fixed.
True, but it is a matter of weighing risks. I can't find it now, but I remember a few years ago there was a news story about how an update to Ubuntu had caused hospitals to start rendering MRI scan results differently due to differences in the OpenGL libraries. For those sorts of use cases, stable is the only option.
I think this is a perfect use case for CentOS/RHEL as opposed to Ubuntu when the machine has only one job and nothing shall stand in its way, ie when you expect everything to be bug-for-bug compatible. But I fail to understand why a vendor of an MRI machine charging tens of thousands for installation/support cannot provide a supported RHEL OS which costs $180-350/yr in the cheapest config [1].
They bought some fancy new computers at work. Our procedures say to use CentOS 7, so we tried it, it ran like shit. Then we reinstalled CentOS 8, same. It worked, but the desktop was extremely slow. After much hair pulling I found the solution: add the elrepo-kernel repository, and update to kernel 5.x
No amount of backporting magic will make an old kernel work like a new kernel.
if you need epel, or quicker life cycles then CentOS Stream should be just fine for you as well
People that run CentOS in prod are normally running ERP systems, Databases, LoB Apps, etc, and the only thing we need is the base distro and the the vendor binaries for what ever is service/app that needs to be installed, and probably an old ass version is JDK...
We need every bit of that 10 year life cycle, and we glad that we will probably only have to rebuild these systems 2 or 3 times in our career before we pass the torch to the next unlucky SOB that has to support an application that was written before we were born...
It's the opposite. Plenty of subsystems in the RHEL 8.3 kernel are basically on par with upstream 5.5 or so, as almost all the patches are backported. The source code is really the same to a large extent, and therefore security fixes apply straightforwardly.
Plus, there are changes (especially around memory management or scheduling) that are fiendishly hard to do regression testing on, so they are backported more selectively.
The upstream for most other packages generally move much more slowly than the kernel. The fast ones (e.g. X11, systemd, QEMU) are typically rebased every other update or so (meaning, roughly once a year).
It also helps that Red Hat employs a lot of core developers for those fast moving packages. :)
Documented cases don't seem to be common, but what comes to mind is the Debian "weak keys" scandal (2008), and the VLC "libeml" vulnerability (2019)[1]
Agreed, the packages in Centos / RHEL are all super old. The RHEL license structure changes all the time and depending on which one you get it may or may not include the extended repos.
Honestly that support is meaningless for some areas I know. In our Data Center we have hit problems with old packages and at the end you will end up with a lot of your own packages. In the end I find Debian to be a good base, and you build the rest by yourself. Even though I use Fedora for Desktop, I always have a feeling Debian is the server choice which I can extend further.
This is false. Debian provides LTS with a 5-years timespan. [1]
And there is even commercial support for Extended LTS now [2]
Also, it's worth noticing that Debian provides security backports for a significantly larger set of packages and CPU architectures than other distributions.
Do you trust Debian LTS? As much as RHEL? The documentation about Debian LTS always made me think it is not a fully fledged thing. I've always felt like Debian releases reached EOL on their EOL date, not their LTS EOL date.
> Debian LTS is not handled by the Debian security team, but by a separate group of volunteers and companies interested in making it a success.
Do you know something I don't? A few years back, Debian changed their LTS policy to 5 years in response to Ubuntu.
> Debian Long Term Support (LTS) is a project to extend the lifetime of all Debian stable releases to (at least) 5 years. Debian LTS is not handled by the Debian security team, but by a separate group of volunteers and companies interested in making it a success.
> Thus the Debian LTS team takes over security maintenance of the various releases once the Debian Security team stops its work.
Arguably, no one should be running a server that long in 2020.
I would say a better reason is that while both are Linux distributions, they are distinct dialects and ecosystems. It isn't impossible to switch, but for institutions that have complex infrastructure built around the RHEL world, it is a lot of work to convert.
It's not really about running servers for 10 years. It's about having a platform to build a product on that you can support for 10 years. RHEL software gets old over time, but it's still maintained and compatible with what you started on.
Consider an appliance that will be shipped to a literal cave for some mining operation. Do you want to build that on something that you would have to keep refreshing every year, so that every appliance you ship ends up running on a different foundation?
> Consider an appliance that will be shipped to a literal cave
This.
A decade ago I was technical co-founder of a company [0] that made interactive photo booths and I chose CentOS for the OS.
There are some out in the wild still working and powered on 24/7 and not a peep from any of them.
We only ever did a few manual updates early on - after determining that the spotty, expensive cellular wasn't worth wasting on non-security updates - so most of them are running whatever version was out ten years ago.
The "don't touch it if it's not broken" philosophy is fundamentally at odds with an internet-connected machine.
You either need to upgrade or unplug (from the internet).
There are still places out there that are running WindowsNT or DOS even. Because they have applications which simply won't run anywhere else or need to talk to ancient hardware that runs over a parallel port or some weird crap like that. These machines will literally run forever, but you wouldn't connect it to the internet. Your hypothetical cave device would be the same.
Upgrading your OS always carries risk. Whether it's a single yum command or copying your entire app to a new OS.
Besides, if you're on CentOS 8 then wouldn't you also be looking at Docker or something? Isn't this a solved problem?
The point is the amount of "touching". Applying security patches to RHEL is still a change, but it's significantly less risky than upgrading a faster-changing system where you might not even get security patches at all for the versions of software you're using unless you switch to a newer major version.
"don't touch it if it's not broken" is not a philosophy, it is a slogan. Some people say it, because it is preferable to them to run old unpatched vulnerable systems rather than spend resources on upgrades. That's just a reality. Some care about up-to-date, some don't. Most people don't really care about security, and some of those don't care even about CYA security theatre. If they did care about security, they wouldn't run unverified software downloaded from the Internet.
Why Docker has anything to do with this discussion?
I think it is mostly about running servers (standard services that don't change much) for 10 years (and more). You don't need 10 year LTS distribution for building a product. You take whatever version of OS distribution you like, secure local copy should the upstream disappear, and vendor it into your product and never deviate from it.
Of course there are use cases, but _ideally_, most workloads are staged, deployed, and backed up in such a way that it is a documented, reproducible procedure to tear down an instance of a server, rebuild, and redeploy services.
And while it may be cumbersome or cause some downtime or headaches if that isn't the case, I find the very need of doing it once every 1-3 years forces your hand to get your shit together, rather than a once per decade affair of praying you migrated all your scripts manually and that everything will work, as your OS admins threaten your life because audit is threatening theirs.
How many simultaneously running machines can you keep updating with this method? If you run non-trivial workloads for hundreds of customers, this becomes high maintenance system already with two machines. It takes ages to upgrade all applications, then validate everything works, then actually migrate with no downtime.
Honestly, 10 years is a long time for a server. I would be honestly surprised if a server lasted 10 years.
But I agree I also get the tone of "servers should be cattle and not pets, just kill them and build a new one". Which can also be done on bare metal if you're using vms/containers. It seems like most people forget these cloud servers need to run on bare metal.
Really? We've colocated our servers for the past 18 or so years.
We have about 40. The oldest is around 17 years old. Our newest server is 9 years old. Our average server age is probably around 13 years old.
The most common failure that completely takes them out of commission is a popped capacitor on the motherboard. Never had it happen before the 10 year mark.
Never had memory failure. Have had disk failures, but those are easy to replace. Had one power supply failure, but it was a faulty batch and happened within 2 years of the server's life.
The last time I worked with a ~8 year old server, it used to go through hard drives at a rate of 1 every 2 months. While we could replace them easily and it was RAID so there wasn't any data loss, I personally would've got fed up of replacing HDDs every couple of months.
Also, most of my experience is with rented dedicated servers and they just give me a new one completely so I never really see if they're fully scrapped.
My read on that is that you should be treating your servers as disposable and ephemeral as possible. Long uptimes mean configuration drift, general snowflakery, difficulties patching, patches getting delayed/not done, and so forth.
Ideally you'd never upgrade your software in the usual way. You'd simply deploy the new version with automated tooling and tear down the older.
I don't get this. If there are many servers, sure. But if it's something that runs on a single box without problem, why on earth should I tear it down?
Also, "running a server for ten years" does not need to mean that it has ten years of uptime. I think that wasn't meant.
If it is connected to the Internet, then I guess the kernel needs to be hot-patched need to be applied to avoid security issues.
Were hot kernel patches available ten years ago? I remember some company who did this (for Linux), and it was quite a while back, so it's possible. But I doubt it was mainstream.
I recall long ago that SunOS boxes had to be rebooted for kernel patches.
"Ideally" - that's the problem. I have half a dozen long tail side projects running right now on Centos 7, and a few still on Centos 6.
Do you have any idea how much effort it is to change everything over to "treating your servers as disposable"?! It's going to eat up a third (to half) of my "fun time" budget for the foreseeable future!
Exactly, young devs here are completely out of touch with operations. Of course ideally something like standard 1TB HDD+32GB RAM system would be upgraded to newer OS and apps version by a central tool in 2hours, but we don't such FM technology yet.
That said, if you're really in the position of depending on a free project for over five years of security support, you probably will be totally fine with just ignoring the fact it's out of support. Just keep running Debian 6 for a decade, whatever. The code still runs. Pretend you've patched. Sure, there are probably some vulnerabilities, but you haven't actually looked to see if the project you're actually using right now has patched all the known vulnerabilities, have you?
(Spoiler, it hasn't: https://arxiv.org/abs/0904.4058)