[otagekki]

Transitioning from CentOS 7 to RedHat 8 and 9 at my former company's private cloud has smooth for most teams, pareto-style, with 80% of migration-related incidents caused by the 20% of the teams that did some really weird changes to the VM's OS that was no longer allowed under RHEL 8 or 9.

At first, I thought it was just to reduce the complexity of managing hardening rules for several OS and OS versions.

[IsTom]

Recently we had issues with RH 9 not having header packages for openssl 1.1 (which means e.g. you can't have erlang < 25). There's a potential for breakage for anything that is 3+ years old.

[bitcharmer]

Not sure if that applies to your line of work but RHEL has been an increasingly annoying source of issues in low-latency space. Their kernels are a bizarre mixture of cherry picked stuff from upstream and a completely bonkers project structure. Also, they break ABI across minor kernel versions which is unthinkable in mainline. What I'm trying to say is: if you're doing more funky stuff with your OS, just go with a distro built on top of mainline.

[eraser215]

Which ABI has red hat broken between minor versions? Can you give some examples that weren't bugs that got fixed?

[p_l]

It's a big issue for companies that have substantial deployment of CentOS7... and FedRAMP or similar clientele.

[bayindirh]

It's a very big issue for scientific clusters which depend on CentOS, too.

We'll find a way, though.

[p_l]

Scientific clusters have more options though, as Rocky/Alma and other alternatives to RHEL are available without dealing with "this distro does not ship FIPS-140-3 certified crypto" or similar.

[bayindirh]

Yes, that's true. However, tons of software was written solely for CentOS, so ironing out the small kinks will take some time.

The problem is, scientific software can silently error out in many ways (like slightly lost accuracy), and detecting and fixing those will take some time.

Considering some of these tools are developed over (almost two) decades, there's a lot of verification we have to go through.

[p_l]

Considering how many interesting issues that specific CentOS7-locked job had just with updating kernel to latest LTS (which required a lot of fun stuff as well, given that it didn't compile with included GCC)...

... I can imagine.

[c0balt]

Rocky and Alma are okay-ish for scientific clusters. They work and drivers appear to mostly cooperate. Some changes, like switching to sssd, were suboptimal but to be expected.

CentOS just had a lot more testing being done by vendors. We still regularly hit problems with, e.g., Lenovo LESI being not aligned with our Rocky systems for many parameters.

[CapeTheory]

Anecdotally, I am hearing scientific computing customers who are unhappy after moving to Rocky - the window where each minor OS version is "supported" is too small and their hardware vendors have trouble staying current. Have seen quite a few move to Ubuntu, and the rest switch to regular RHEL.

[worthless-trash]

Talking firsthand, There is significant effort into ensuring that both 8.6.z (through to current) and 9.2.z (through to current) are fedramp compliant, this requirement eats my day, every day.

8.6 and 9.2 kernels are released more frequently than other streams, if you want more frequent updates for compliance reasons, these are the streams to use.

[tiberious726]

If you were using centos for fedramp, unless you got a variance from the feds, you were not in compliance. No one actually paid to have NIST evaluate centos's binaries (evaluation/certification must be of the compiled binary, not the source code, barring an exception made for openssl, and only openssl, not the kernel)

[p_l]

This wasn't done yet for FedRAMP High, but it passed the audit for earlier customers demanding FIPS-140-2 compliance.

Not that I am not saying it wasn't a clusterfuck of epic proportions in general.

[Twirrim]

Why is it a problem with FedRAMP? CentOS 8 is FIPS certified, has STIG profiles etc.

[p_l]

Last I checked, CentOS Stream is not FIPS certified, and CentOS 8 is already past EOL, which makes it not allowed for FedRAMP.

And IIRC CentOS8 FIPS certificate was taken out by Red Hat (wouldn't have had to implement our own FIPS handling on CentOS7 if not for that move).

[tiberious726]

Centos was never FIPS certified https://csrc.nist.gov/projects/cryptographic-module-validati...

[p_l]

There was a time when RHEL FIPS certificate also applied to CentOS, and one of my former $DAYJOBs depended on that for a long time. Then Red Hat pulled the cert, and later there was a mad scramble to get rid of CentOS because of the EOL :)

[tiberious726]

Oh wow! Do happen to have a link to the old cert? I /really/ looked for a way for us to stay on centos when we started doing this kind of stuff.

[Twirrim]

Doh, sorry. Forgot it only hit Red Hat 8 (and derivatives).