[em500]

Maybe it's a bit too obvious to spell it out, but

1. Red Hat Inc. does not want people to build and/or distribute gratis RHEL8 or clones. It would be trivial to just put the actual RHEL8 iso as an unsupported download on their ftp/www server and sell the support separately, like Oracle or Canonical do. Instead, they kept this ridiculous make work project called CentOS around, that involved non-trivial manual labor meticulously rebranding RHEL into RedHat owned CentOS-the-laggard-RHEL-clone, whose users apparently mainly value it for being as close to RHEL as possible without paying. To a distant observer, the whole setup looks quite absurd. I think it's actually good that they finally put an end to it. But they should have either not started CentOS 8 at all or rode it out till the end. Pulling the plug at 20% of it's lifetime is plainly a shitty move.

2. It appears that building a legal RHEL8+ clone is quite a task, and I'm guessing the amount of work involved is largely controlled by Red Hat Inc. I.e., they can make running/maintaining a project like Rocky or others more and more expensive if they choose to. I believe they going to test the limits of how difficult they can make building from their sources within the boundaries of the GPL. If you think I'm wrong, just put up a mirror of the RHEL8 (not CentOS8) SRPMs and see how long it stays up. Clearly they're not acting in the spirit of the GPL, even if they are in the letter.

3. Given the previous points, I believe any project like Rocky is a losing proposition. If the "community" really values enterprise stability so much, better put the effort into an extra-stable-extra-LTS fork of Debian or even Ubuntu, and preparing for transition away from RHEL. Or just pay up, if you really believes the RHEL stability is so valuable. Clone projects are by necessity extremely dependent on actions of Red Hat Inc which has largely opposite interests. I don't know why people would volunteer for that.

[dralley]

> If you think I'm wrong, just put up a mirror of the RHEL8 (not CentOS8) SRPMs and see how long it stays up. Clearly they're not acting in the spirit of the GPL, even if they are in the letter.

The GPL doesn't require you make the sources public to everybody, just the people to whom you are distributing your software.

But Red Hat does provide sources, to everybody. They go above and beyond what the GPL actually requires.

[em500]

The ftp site does not contain sources for RHEL 8 (it's just sources for a bunch of add-on packages). The RHEL 8 sources are in git.centos.org, though not in SRPM form.

I know that GPL only requires Red Hat to provide sources on request to those receiving their binaries. But in principle the receiver can then legally redistribute those sources. Now AFAICT nobody's actually doing that redistributing. I don't know whether that's due to lack of interest, or due to some Red Had shenanigans that makes redistribution illegal/unattractive.

[mbreese]

RedHat has to permit the redistribution per the GPL. But there is nothing stating that your support contract can’t be cancelled if you do it. (I don’t have any first hand knowledge of this, just a guess).

[bigbubba]

Grsecurity also uses this 'loophole.' Seeing this scheme go mainstream is really disheartening; I feel that it really undermines the intent and social value of the GPL.

[ikiris]

It's supposed to be against the GPL, but no one is willing to chase it.

[dralley]

In what way is it "supposed to be against the GPL"?

[type0]

> I don't know whether that's due to lack of interest, or due to some Red Had shenanigans that makes redistribution illegal/unattractive.

Red Hat is overly protective of their trademarks, and this includes CentOS, Fedora, CoreOS etc

[kakwa_]

Also, RHEL is packaging pieces of software that are not under GPL/LGPL. With permissive licenses, they could probably heavily restrict source redistribution and availability.

If half the userland is not available as source (patch and packaging included), a CentOS-like project would not be possible.

Effectively, if RedHat/IBM wanted, there are a lot of dick moves which could effectively kill Rocky.

[coldtea]

>But Red Hat does provide sources, to everybody. They go above and beyond what the GPL actually requires.

If you had been around in the late 90s, their lip service was that they were far more than just "providing sources to everybody" (as if that's some feat), and far more about the spirit of GPL and the triumph of FOSS than what the GPL letter requires...

Of course that was lip service, like Google's "Don't be Evil", but don't go around pretending Red Hat is some benevolent "above and beyond GPL" entity...

[p1necone]

> better put the effort into an extra-stable-extra-LTS fork of Debian or even Ubuntu

I've always thought of Debian LTS as basically the end all be all of Linux server OS stability (let's ignore the BSDs for this exercise), is the main draw of CentOS over that just the longer LTS period or is it also meaningfully more stable?

[bigbubba]

From what I understand, it's not just a matter of longer support, but also training. A lot of companies have/had mixed RHEL/CentOS environments, with RHEL on the machines they really want RH support for and CentOS on everything else to save money. Having a mixed RHEL/Debian environment would probably be a pain in the ass for all your sysadmins.

[garyrichardson]

On top of this I also believe that companies prefer the RHEL opinionated implementation of a linux system vs Debian's.

RedHat is a company that sells to other companies. Therefore it has to implement their linux and make decisions in a way that works well with how other enterprises think. Big companies aren't comfortable depending on "the community" to do the right thing for them.

[johannes1234321]

On top of all these companies want to run (commercial) software from other vendors, which sometimes is certified only on RHEL.

[joana035]

Not really, I have seen CentOS based infrastructure having more problems (xfs breaking docker, frankenkernel 3.10 in 2020?) whereas with Debian things just works and are not heavily patched to the point of breaking ABI.

[Nullabillity]

In a vacuum, absolutely. I think GP's point was that if you already have to run the frankenkernel on some machines (because of support), then it makes management easier if you at least run the same broken frankenkernel on all of them.

[bionsystem]

I've seen issues with debian as well in production systems, from the packaging systems most notably. Both distros are mostly good with some flaws.

[jhatemyjob]

The way you describe it reminds me a lot of the iOS jailbreak community. There were some people that thought that it could overtake vanilla iOS but they failed to realize they are perpetually fighting directly with a large corporation and will probably burn out and lose. It sucks but that's reality

[evol262]

> 1. Red Hat Inc. does not want people to build and/or distribute gratis RHEL8 or clones. It would be trivial to just put the actual RHEL8 iso as an unsupported download on their ftp/www server and sell the support separately, like Oracle or Canonical do. Instead, they kept this ridiculous make work project called CentOS around, that involved non-trivial manual labor meticulously rebranding RHEL into RedHat owned CentOS, whose users apparently mainly value it for being as close to RHEL as possible without paying. To a distant observer, the whole setup looks quite absurd. I think it's actually good that they finally put an end to it. But they should have either not started CentOS 8 at all or rode it out till the end. Pulling the plug at 20% of it's lifetime is plainly a shitty move.

But [they do](https://developers.redhat.com/products/rhel/download). A subscription buys you updates and support. RHEL itself is free. In addition, considering that almost all of Red Hat's products are "upstream first", branding is generally a single additional RPM which changes some colors. It is _not_ hard to rebrand RHEL

> 2. It appears that building a legal RHEL8+ clone is quite a task, and I'm guessing the amount of work involved is largely controlled by Red Hat Inc. I.e., they can make running/maintaining a project like Rocky or others more and more expensive if they choose to. I believe they going to test the limits of how difficult they can make building from their sources within the boundaries of the GPL. If you think I'm wrong, just put up a mirror of the RHEL8 (not CentOS8) SRPMs and see how long it stays up. Clearly they're not acting in the spirit of the GPL, if they are in the letter.

The only case in which this has happened is changing the packaging of the kernel-sources SRPM so Oracle could not pick and choose particular patches, and instead had to deal with a single massive diff to make stuff like kpatch work. The "expensive" part of building an EL8 clone is standing up a bunch of Koji builders. That isn't necessary either, strictly. It just makes "turn this SRPM into an RPM and combine it with a bunch of others into a temporary repo we can dogfood/release" easier. As always, it comes down to cost.

> 3. Given the previous point, I believe a project like Rocky is a losing proposition. If the "community" really values enterprise stability so much, better put the effort into an extra-LTS fork of Debian or even Ubuntu, and preparing for transition away from RHEL. Or just pay up, if you really believes the RHEL stability is so valuable. Clone projects are by necessity extremely dependent on actions of Red Hat Inc which has largely opposite interests. I don't know why people would volunteer for that.

The problem with this, broadly, is that the "community" is comprised of a lot of Red Hat-employed engineers. This idea that Linux is a bunch of "community" people that RH/IBM/whomever siphon off is completely fallacious. Sure, they exist, but the vast majority of Linux development is commercial. Some bug is reported in a downstream product that a customer pays for, or a feature is requested by a major customer (or the technical debt involved in maintaining something gets too high, or whatever), and professional engineers who are being paid to work on it write the patches upstream. Once they're merged, they get picked downstream into some kind of productized version.

Given that most of that happens in Fedora (or OKD, or oVirt, or whatever upstream is for a given product), CentOS being run by RH was pretty much the same embrace+extend+extinguish philosophy. The vast majority of RH engineers run Fedora, or an EL clone, and that isn't gonna change. What's different is that RHEL is no longer a growing revenue stream, and CentOS users didn't convert to RHEL at a large rate (or they reported CentOS bugs against RHEL, which is strictly against Red Hat's support policy).

What customers values is indemnity and the ability to point their finger at someone external, plus normal stuff like a security response team and responsible+timely disclosure during major CVEs. What the community value(d|s) is the ability to run commercial software that the vendor supports on RHEL without actually _paying_ for RHEL. It's not that they value 'enterprise stability'. It's that they value being able to run SAS or whatever without hacking the hell out of the installer. That isn't offered by an extra-LTS fork of Debian or Ubuntu, and no amount of complaining on Hackernews is gonna change that.

[l2dy]

* The free download is only "for Development Use".

* Debranding is much more work than a single RPM. https://wiki.centos.org/About/Building_8

    No you can't just "sed s'/Red Hat/CentOS/' (someone always offers that)
    There are times when you do replace and times when you don't.

[evol262]

That isn't how Red Hat does things in most of their projects. oVirt/RHV provides a reasonably "standard" example with their "rhevm-branding-rhev" package: http://ftp.redhat.com/pub/redhat/linux/enterprise/7Server/en...

It's a single package with relatively few additions, mostly color schemes in CSS.

Anaconda documents this in /usr/share/branding

RHEL puts a lot into subpackages of redhat-releasea

But it isn't really the point. There are thousands of packages. Only a handful actually deal with branding. I was an engineer at Red Hat for 7.5 years, and I left last June. I'm intimately familiar with this, and while it's more complex than sed, it's much simpler than it sounds, and the CentOS people are definitely familiar with it. Branding is NOT their hurdle. It's the build system, as they repeatedly mention.

And that is Koji, which is also open source: https://docs.pagure.org/koji/

You're blasting Red Hat for making it seem like they're somehow opposed to clones. That's fatuous. A major engineering company with complex workflows uses their own build system. News at 11. They also make this free and document the hell out of it. Anyone who has ever built a package for Fedora is familiar with it. Anyone who has ever dealt with release management/engineering for a product inside Red Hat is extremely familiar with it, and that includes many of the core CentOS team, plus the Fedora RelEng SIG is easy to join if you want to learn. It's complicated, but it's not black magic or even hard information to get. CentOS ALREADY USES IT.

It really doesn't matter whether the RHEL image is "for development use only" (you didn't want support anyway). Stop moving the goalposts.

[chillfox]

I think one of the reasons people don't really value "enterprise stability" is because it is mostly a myth.

[corty]

Agreed. The reason is more "we are running supported software in a supported combination of hardware, OS and application". That's what auditors are looking for. I've yet to see a real benefit beyond the paperwork advantage.

[em500]

> But [they do](https://developers.redhat.com/products/rhel/download). A subscription buys you updates and support. RHEL itself is free.

That's true in a literal-but-useless sense. Developers-RHEL might be bit-identical to real-RHEL at some dates, but as a product it's of course very very different due to lack of interim updates. And that's aside of the smaller speedbumps of registration and having to involve Legal if you want to run in a company.

> In addition, considering that almost all of Red Hat's products are "upstream first", branding is generally a single additional RPM which changes some colors. It is _not_ hard to rebrand RHEL. [...] The "expensive" part of building an EL8 clone is standing up a bunch of Koji builders. That isn't necessary either, strictly. It just makes "turn this SRPM into an RPM and combine it with a bunch of others into a temporary repo we can dogfood/release" easier. As always, it comes down to cost.

I don't know how true that is, the CentOS wiki makes it sound like like the debranding part is non-trivial/manual labour intensive. So according to you the main bottleneck is hardware / buildservers? If that's true, one wonders why even the minor CentOS 8.x releases lag RHEL 8.x by 4 to 6 weeks.

I guess we'll get to see it soon, with Rocky Linux development.

> The problem with this, broadly, is that the "community" is comprised of a lot of Red Hat-employed engineers. This idea that Linux is a bunch of "community" people that RH/IBM/whomever siphon off is completely fallacious. [...]

Sure, I largely believe the long-running narrative that a lot of core Linux development is paid/done by Red Hat, and that most other distros, including Debian/Ubuntu are free-riding to some extend. That's why I not really behind the cheering for Rocky, and think it's fairer in the end to either pay up, or roll up the sleeves and do a real _community_ enterprise OS based on Debian instead of cloning RHEL.

> What customers values is indemnity and the ability to point their finger at someone external, plus normal stuff like a security response team and responsible+timely disclosure during major CVEs.

The first part might be true of RHEL customers, but I don't think it's true for CentOS customers. The customer base is of course diverse, but my guess is that for a very large part of the (CentOS, not RHEL) users, objections to moving to Debian/Ubuntu are practical (legacy/switching costs, maybe followed by maybe proprietary software support), much more than principled/legal.

[op00to]

Developers RHEL is RHEL. You get access to the same updates as a paid RHEL subscription. Stop spreading FUD because someone moved your free cheese.

[strenholme]

>>roll up the sleeves and do a real _community_ enterprise OS based on Debian instead of cloning RHEL<<

I would love to see this too, but backporting security patches is not the kind of “sexy” programming we can motivate programmers to do “for fun and for free”, especially when we tell them to do it for X number of years without having a paycheck to motivate them.

[evol262]

> That's true in a literal-but-useless sense. Developers-RHEL might be bit-identical to real-RHEL at some dates, but as a product it's of course very very different due to lack of interim updates. And that's aside of the smaller speedbumps of registration and having to involve Legal if you want to run in a company.

Again, stop moving goalposts. Your comment was "why can't they just put it on FTP/WWW without support". That's exactly what they do. You aren't required to register it in any way, including downloading. You want a free "product". That isn't their business model. At least they make the sources for everything available, which took Canonical years for Landscape.

> I don't know how true that is, the CentOS wiki makes it sound like like the debranding part is non-trivial/manual labour intensive. So according to you the main bottleneck is hardware / buildservers? If that's true, one wonders why even the minor CentOS 8.x releases lag RHEL 8.x by 4 to 6 weeks.

This is literally the build system. I have real-life experience branding a number of Red Hat products, including rebranding Anaconda and the base system. See for example https://github.com/oVirt/ovirt-release/blob/master/ovirt-rel...

The lag is because Koji (which is also used internally) uses a hierarchical build/tag system. See https://docs.pagure.org/koji/using_the_koji_build_system/ and https://docs.pagure.org/koji/tag_inheritance/

For a base release, a mock root needs to be bootstrapped. This is more or less by hand, and does definitely require incremental builds of and the rest of the base system. I guarantee the CentOS releng team has scripts that do this, but I don't know where they'd keep them (I never worked directly with that team). Once they're in a koji buildroot, that buildroot can be used to bootstrap its way to a 'release' buildroot, with successive tags. Again, this is something that any build engineer should be familiar with.

The real wrench with 8 is modularity, which is also present in Fedora. If you really want to help CentOS go help them fix it:

https://lwn.net/Articles/805180/

Issues like "special version of RPM in the buildroot" are weird non-issues specifically related to how Koji works, but Modularity does have real problems.

> Sure, I largely believe the long-running narrative that a lot of core Linux development is paid/done by Red Hat, and that most other distros, including Debian/Ubuntu are free-riding to some extend. That's why I not really behind the cheering for Rocky, and think it's fairer in the end to either pay up, or roll up the sleeves and do a real _community_ enterprise OS based on Debian instead of cloning RHEL.

This is a no true Scotsman argument. I don't know what isn't "real" or "community" about CentOS other than the fact that there was/is overlap between CentOS maintainers and RH employees. Just because they have day jobs doesn't mean they can't be part of the community, too. Hand wringing about what's fair defeats the purpose of using a distro like a tool to accomplish goals.

> The first part might be true of RHEL customers, but I don't think it's true for CentOS customers. The customer base is of course diverse, but my guess is that for a very large part of the (CentOS, not RHEL) users, objections to moving to Debian/Ubuntu are practical (legacy/switching costs, maybe followed by maybe proprietary software support), much more than principled/legal.

You inverted this argument and you're asking the wrong questions. The question isn't "why aren't people moving off CentOS?", it's "why did they use CentOS in the first place?"

It's because they were already familiar with RHEL from previous jobs, and wanted familiar tooling (apt may be nicer than yum was, but dpkg is a dumpster fire for package maintainers compared to RPM, and RPM's tooling is much more cohesive than digging around in 10 different manpages for apt-cache || apt-file || dpkg -L || whatever to get information). Kickstart is nicer in many ways than preseed. Sure, the costs of swapping all of that are non-trivial both in the time investment for administrators to rewrite tooling and for the marginal loss in productivity until they re-learn tooling.

Going forward, container-first distros are getting the brunt of new deployments, which is exactly why RHEL isn't a growing revenue stream for Red Hat anymore, and why it doesn't make sense to keep pouring money into CentOS.

Red Hat is focused on OpenShift/OKD as the new "platform". Containers have their place, and it isn't everywhere for lowly end-users, but RH didn't drop CentOS to fuck over the community. They reduced support for the community because RHEL (and CentOS) are increasingly irrelevant to a company which has put their eggs in the CoreOS+Openshift basket as their future. That isn't specific to them.

All of the major vendors see the writing on the wall. What's better than making your systems easy to manage? Making it so they don't need management at all. Do everything in k8s and update a single system image quarterly (or whatever), and leave traditional workflows as an afterthought. Red Hat is lucky enough to already have RHEL, but if I were starting a new for-profit Linux company in 2020, I'd do exactly what CoreOS did or Rancher does.

[em500]

> Again, stop moving goalposts. Your comment was "why can't they just put it on FTP/WWW without support". That's exactly what they do.

That's exactly what they don't. You're the one who innocuous altered that to "without support and updates" which you know very well makes all the difference. If CentOS had the same non-update schedule as developer-RHEL, approximately nobody would use it in production. Contrawise, if RHEL did provide timely yum/dnf updates (and remove the murky "Development Use" clause), everyone would run that instead of CentOS. So now you're telling me to stop moving the goalposts back after you moved them to another field?

> You aren't required to register it in any way, including downloading.

Maybe I'm a bit dense, but I clicked on every download link on your page and they all led me to "Log in to your Red Hat account". Maybe you can post a direct URL to the ISOs?

> You want a free "product". That isn't their business model. At least they make the sources for everything available, which took Canonical years for Landscape.

No, I don't want a free product. I was stating from the first post that Red Hat Inc don't want people to have that. Which I'm completely fine with and fully support. You seem to be in complete agreement, so I don't know why you bothered with some quasi-rebuttal in the form of freebie Developer RHEL link which is very different from the CentOS/Rocky proposition.

> [details about the build process]

Ok, maybe the debranding part is not a big deal, I don't know. But my point only rests on the process being labour-intensive, hence expensive. Which I don't know you're disputing or agreeing with. If a CentOS rebuild is necessarily labour-intensive, I don't see a bright future for Rocky or similar projects. I think it is, since AFAICT almost every clone except Oracle gave up trying to keep up with RHEL 8. It's hard to prove things either way, but we'll find out soon enough if we follow the Rocky project.

> You inverted this argument and you're asking the wrong questions. The question isn't "why aren't people moving off CentOS?", it's "why did they use CentOS in the first place?" It's because they were already familiar with RHEL from previous jobs,

So inertia / legacy / switching costs, we're exactly in agreement.

> and wanted familiar tooling (apt may be nicer than yum was, but dpkg is a dumpster fire for package maintainers compared to RPM, and RPM's tooling is much more cohesive than digging around in 10 different manpages for apt-cache || apt-file || dpkg -L || whatever to get information). Kickstart is nicer in many ways than preseed. Sure, the costs of swapping all of that are non-trivial both in the time investment for administrators to rewrite tooling and for the marginal loss in productivity until they re-learn tooling.

Now this is a completely different argument, that the RHEL/CentOS tooling are intrinsically superior to Debian/Ubuntu's. I'm pretty skeptical, since it implies that organizations who do run Debian/Ubuntu could save a lot by switching to CentOS and a bit of retraining. But this is of course not a dispute that's going to be settled in a thread like this, so let's leave it at that.

> [something about containers, OpenShift, K8s, divining Red Hat's Grand Strategy]

This does not seem on topic, so no comment.

Maybe I just expressed myself badly, so let my try again. I predict that Rocky Linux will not be a big success, and don't think doing a free-beer RHEL clone (CentOS as most users understood it) is a worthwhile endeavor, despite a seemingly large audience. Yes, giving good stuff away for free is popular (and I do believe RHEL is a good product). Because they're largely dependent on RH, who 1) appear not very enthusiastic about the idea of people running RHEL for free even without support, 2) can largely determine how expensive running a clone project will be. That's why IMO it's better to analyze if users really need a strict RHEL clone, or if what they want from it (xLTS, better tooling, commercial software compatibility, whatever) could be better developed on top of Debian, whose incentives seem to class less. People who really really need a RHEL clone: time to pay up or go with the Stream (it probably really not-so-bad).

Now unlike you I have zero inside knowledge or experience, so maybe I'm just talking out of my ass. But I am willing to make a somewhat falsifiable prediction, that Rocky and similar clone projects don't have much chance of success. Maybe I'm all wrong and Rocky can with a handful of volunteers and some clever scripts, resurrect CentOS-as-people-understood-it. Or maybe some deep-pocket 3rd party with a better brand than Oracle will step up. We'll find out a year or so.

[evol262]

> Maybe I just expressed myself badly, so let my try again. I predict that Rocky Linux will not be a big success, and don't think doing a free-beer RHEL clone (CentOS as most users understood it) is a worthwhile endeavor, despite a seemingly large audience. Yes, giving good stuff away for free is popular (and I do believe RHEL is a good product). Because they're largely dependent on RH, who 1) appear not very enthusiastic about the idea of people running RHEL for free even without support, 2) can largely determine how expensive running a clone project will be. That's why IMO it's better to analyze if users really need a strict RHEL clone, or if what they want from it (xLTS, better tooling, commercial software compatibility, whatever) could be better developed on top of Debian, whose incentives seem to class less. People who really really need a RHEL clone: time to pay up or go with the Stream (it probably really not-so-bad).

> Now unlike you I have zero inside knowledge or experience, so maybe I'm just talking out of my ass. But I am willing to make a somewhat falsifiable prediction, that Rocky and similar clone projects don't have much chance of success. Maybe I'm all wrong and Rocky can with a handful of volunteers and some clever scripts, resurrect CentOS-as-people-understood-it. Or maybe some deep-pocket 3rd party with a better brand than Oracle will step up. We'll find out a year or so.

So, let's try this. I agree that Rocky will not be a big success, but completely because I don't think there's a major market demand for a RHEL clone in 2020. New deployments will end up being a container Linux distro with deployments inside VMs or containers. (RH)EL8 is relatively new with low-ish adoption. We still had customers on EL6 six months ago when I left. Institutional customers aren't likely to move to EL8 until EL7 is nearing the end of phase 3 support, since the costs involved in reworking their applications to work (and their build/config management toolchain to work with modularity+etc. This has nothing to do with Red Hat's stance, which was (for the 7.5 years I was there, and still, from everyone I know who's still there) very overtly pro-upstream.

Users don't need a strict RHEL clone. They need a life raft until they can move to containers. It's not worth re-training administrators how to use some other package ecosystem. Rocky will fail, but for these reasons

[op00to]

> 1. Red Hat Inc. does not want people to build and/or distribute gratis RHEL8 or clones.

Why does Red Hat release everything as source RPMs, and not, say, one big tarball of sources which would still meet the source requirement of the GPL?

[dcdsc]

Red Hat stopped long ago of being an open source company with open source values unfortunately, not to mention that a lot of good talent had exodus long ago to FANG companies which are now driving most of development.