My team in Red Hat’s Open Source Program Office provides hosting and sysadmin care to various open source communities, mainly using CentOS Linux, so when the CentOS Project announced that CentOS Linux would be discontinued in favor of CentOS Stream, the move was a real concern for us.

We had many uncertainties after the announcement, which was a bit abrupt. Rich Bowen, the CentOS Community Manager at the time, clarified the positions of Red Hat and of the CentOS Project regarding the changes. For one thing, we learned that new, “CentOS Classic-style” community distributions were expected (and welcomed), so we could wait for one of those to emerge if CentOS Stream didn’t meet our needs.

We had to decide whether to remain with CentOS, by shifting to CentOS Stream, to choose a new, forked distribution, or to switch to Red Hat Enterprise Linux.

We’ve used CentOS Linux for our services because it’s a slower-moving distribution than something like Fedora Linux. Since CentOS Stream is the upstream for the next minor release of Red Hat Enterprise Linux, we had reason to expect the changes to remain slow-moving. In addition, the fact that CentOS is now positioned upstream of RHEL gives us the opportunity to contribute bug fixes or feature requests to the project.

After more research and discussion, it became clear that we would not know for sure whether CentOS Stream would work for us without testing. We started experimenting first with test VMs, then new projects, followed by migration of non-critical machines, to prove out whether we should make CentOS Stream 8 our default.

Evaluating Changes

Switching to Stream required a few changes in our deployments, so let’s document them as well as noticeable differences we had to deal with.

VM Installation

We use PXE (DHCP+TFTP) for hardware installation and libvirt for managing VMs. We then use Ansible to deploy VMs.

To install Stream directly it’s simply a matter of changing the version: “8” to “8-stream” but unfortunately our Ansible rules were using integer comparisons to be able to adjust the path of the netboot files in the repository depending on the version. We modified the code to extract the number at the start of the string before comparison.

We also use the mirrorlist to ensure switching to a working mirror at install time. The URL contains the release version in a variable ($releasever) but for stream this does not include the “-stream” suffix and it surprisingly resulted in installing the non-stream version. This was discussed on the centos-devel list. We adapted the role to use our Ansible variable directly instead of $releasever.

PXE Installation

We started sharing distribution, repository information as well as some logic to generate the kickstart/preseed files between the VM installation role and the pxe role) in a dependency role but the work is unfinished and we had to port the previous adaptations into these roles.

Ansible roles

All roles use ansible_distribution_major_version which is always numeric to change behavior depending on the OS version and therefore no adaptation was needed.

We also test our roles using Molecule for our main deployment OSes and we already had started adding CentOS Linux 8 as target along CentOS 7 to catch problems and adapt to the new major version. Switching to Stream did not involve any specific changes.

Switching existing hosts to Stream

Once we had our deployments updated to support Stream we wanted to migrate all EL8 deployments to Stream automagically.

We had to remove old repository configurations that were in the way and then we simply used “dnf swap”.

Conclusion

We started using CentOS Stream in April 2021 in our infrastructure and made the few adaptations needed. In June we added the automatic switch to Stream and that went seamlessly. Now except for specific needs we use 8 Stream by default. For one thing, we still have VMs on EL7 because migration takes time, but this is due to software upgrades, not to the Stream changes.

Unless you have very specific needs most users just won’t feel the difference. Changes to your infrastructure deployment rules should only affect OS installation and low level settings. Updates, instead of coming via point releases, come as they are ready and reach you faster, which seems like a nice-to-have feature. In fact this is very similar to stable updates in Debian (https://wiki.debian.org/StableUpdates) or other distributions but expanded to the whole distribution.