CentOS Hysteria

Firstly, I do not work for CentOS, Red Hat or any other Linux distribution. Nor am I particularly attached to CentOS specifically. I use various different flavours of Linux for desktops, laptops and servers I own/manage.

There have been many posts (and a lot of hysteria) around the recent CentOS announcement: CentOS 8 will no longer be binary compatible with Red Hat Enterprise Linux 8 (RHEL 8) with dozens of people quick to investigate alternatives and claim they are jumping ship.

In reality, the change to CentOS is likely to affect only a small number of people using CentOS for servers (this doesn't apply to those running it on desktops and workstations, I'll come onto why later on...)

Ten years ago, I may have fallen into the other camp but the way we use and configure servers has changed significantly since then.

What is CentOS Stream and why is it being labelled as bad?

Before getting into details, let me outline the issues people have with CentOS stream:

1. It's rolling release

Rolling release is bad for a server... or at least conventional wisdom would suggest so. Here's the potential problem: When you run dnf update, php or some other package jumps from verion 7.x to 8.x and there's a chance your whole website breaks.

Scroll down and I'll make the case that rolling release for servers is a good thing these days.

2. It is no longer binary compatible with RHEL 8.

If you get an RPM file for RHEL8 it may (quite probably) won't work on CentOS Stream 8. You'll need a different package compiled specifically for CentOS Stream.

Who does this affect negatively?

Although I'm making a case for sticking with CentOS despite the changes there is one group of people who it will affect.

The people who have lost out here and are rightfully upset are software developers who write applications for Red Hat by building them on CentOS. Until now they could use a CentOS development/testing machine (reducing their licencing costs) and then ship the software to RHEL clients, safe in the knowledge that their RPM will install and run just fine.

If you're developing software that you are compiling and packaging for use on RHEL then the changes to CentOS has potentially made your life a lot more difficult and for those people I do strongly recommend looking at alternatives.

If the software you are developing isn't compiled (or is JIT compiled: Node.js, PHP, Python, etc) then it's not going to make any difference to you. Similarly, if you're just running a server and not developing software specifically for Red Hat based distros then you don't really have anything to worry about.

Why it makes no difference for your servers

I'm running dozens of different types of server across multiple physical CentOS installations. Web servers, mail servers, databases, half a dozen game servers, NextCloud, Bitwarden, the list goes on.

The thing is, not one of them is installed directly in CentOS using dnf. In fact, the only native CentOS packages I have installed are the base OS packages (kernel, systemd, etc), firewalld, openssh and docker.

All of those different servers and applications I'm running on my CentOS machine are being run in Docker. The advantages of this are far too long to get into here but one of them is that the underlying OS barely matters. The application doesn't know or care whether it's running on CentOS, Arch, Debian or Docker for Windows.

The rolling release issue I described earlier doesn't exist when all your software is using docker images. Each application (each website, for example) can control exactly which PHP version it's using and be upgraded at different intervals.

CentOS can make as many drastic changes to their packages as they like. We are in an age where the underlying operating system in use makes zero difference to the applications running on it. As long as I can run a firewall, docker and get timely security updates for the kernel and openssh, nothing else about the OS makes the slightest difference to the way my applications run or I manage applications on the server.

The case for a rolling release server distro

If your server is running Ubuntu, CentOS 7, RHEL 8 or any versioned distribution, there will come a time when you have to do a dist-upgrade, taking you from one version to the next. Avoiding doing this will prevent you from getting important security updates. This may be years down the line, but it will happen eventually.

When this happens, you run into the exact problem I mentioned earlier: PHP or some other package has a major version bump and there's a chance your websites stop working.

If you have 20 websites running on the server, that's 20 websites code to audit before doing the upgrade. For anyone who's ever done this, it is a stressful experience.

A distribution version upgrade requires you to be sure all your applications (Websites, server configuration, etc) work on the new version. It quite often requires a lot of configuration changes, depending on what the server is doing. Server config files get moved around or replaced. It is not a task that a server admin looks forward to.

Docker, thankfully, has generally solved all these problems. You can upgrade the PHP version for each website one at a time, configuration file locations aren't going to change and upgrading the server distribution introduces very few headaches because none of the application code is ever exposed to the changes that come with a global upgrade to packages on the server.

We do want our sever to have the latest kernel security patches and updates to openssh. With all our applications running in docker, it's safe to make as many changes to the underlying OS as you like as the application will never know the update occurred.

The conventional wisdom that a rolling release distro for a server is a bad idea is out of date because the way we manage servers has changed. The kind of issues that a rolling release potentially introduces don't exist when your applications are containerized.

Of course running this way you do have to ensure that you periodically update the PHP image your containers are using, but that's no different to upgrading PHP on the server. The difference is that you can perform the upgrade one website at a time instead of globally.

Conclusion

If you are running services like NGINX, PHP and MySQL installed globally on your server through its package manager (dnf, yum, apt-get, pacman, etc) look at docker, there are far more advantages than I can list here.

For those of us already running all our services in containers, this CentOS announcement makes zero difference. In fact, jumping to a rolling release for our web servers is mildly convenient.

It is probably worth using this opportunity to explore alternatives, but I don't think there is any reason to immediately jump ship like many others have suggested.