There seems to be a new trend in IT -- immutability. But what is it? Immutability is a security mechanism intended to assure the user (or engineer) that the system they built remains as it was built without the addition of any mailware or modified configurations caused by any nefarious actors. Think of it as installing your operating system on a CD. A bad actor cannot modify the files on the CD because they are read only. This is the same for the operating system partition. But how do you configure a system to be immutable? It is much easier than you think, but it does come with its limitations.
There is no simple way to build a server infrastructure with immutable servers. Because of the complexity, I only recommend building immutable desktops at this time. Most server installations go through the motions of allowing you to select partition settings, and which packages to install to a system as it is being deployed. With immutability, storage needs to be configured in such a way that the operating system partition is locked as read only and all user data is written to a separate partition. This in general would prevent updates from ever being applied. It would also prevent software from being installed and configured. To get around that, software is installed in a manner that it is containerized.
Several distributions have met the challenge by providing specialized ISOs that allow you to deploy an immutable system. Enter Fedora 40 Silverblue as one such distro. I learend about this release from Paul Hill over at Neowin with his review and discussion of the release.
Security does not start after a system is installed. It starts before you download the installation media and use it to build a base system. Installation images are not immune from tampering as evidenced by the swapping out of Linux Mint ISOs for those with a backdoor installed to create a botnet on installation way back in 2016. (More information on this attack can be found here.). In short, the attacker replaced the installation ISO in the master repository with one that contained a backdoor enabling his botnet. Hundreds of users (quite possibly more) downloaded the ISO and built systems from it without ever realizing they were duped into installing malware. To combat this, step one is to ensure that the ISO you obtained is from a legitimate and trusted source.
Fedora provides a powerful way to accomplish validating your installation by using the Fedora Media Writer to download only verified copies of the installation media. This requires a 32GB USB thumb drive or similar media from which you can boot. It provides all of the security validation on the backend in building the USB to ensure that the data written to the USB is the data from the legitimate source of truth for the OS. It is the recommended way to download and install the desktop image as it will handle the download and signature validations on its own and save you all of the manual effort. It will also create the USB as a ready-to-go installation that you can later install to the hard drive of the machine if you want to.
A less hands-off approach would be to download the ISO directly from the fedoraproject.org website. If you choose to go this route, I cannot urge you strongly enough to go through the steps provided for each ISO to verify the download. This checks that the signature on the file is verified and that the image has not been tampered with. Yes, it seems excessive. As Linux Mint users learned, failure to perform this simple validation could leave you in a not-so-desirable state with a system that was hacked before it was ever even powered on. Channel your inner Nike and just do it. You have nobody but yourself to blame if you don't.
Silverblue makes itself immutable by blocking write access to the root / directory. The /usr directory and everything underneat it can only be mounted as read-only. All files created by the user including system logs are stored in /etc and /var and are writeable. Everything associated with the base system is therefore locked and identical between machines. There is no more configuration drift between machines. Every one within the same version is identical.
How do I run additional software? Silverblue runs all new applications as containers using Flatpaks. This makes the application run in isolation. The container has everything needed to run the application. When the application is closed, the container is destroyed. Since each container is a sandbox, there is no need to worry about them modifying system settings. They are quarantined from the base system.
But what about security updates? We all know that new vulnerabilities are discovered every day and patches are released to mitigate these issues. If the operating system is non-writeable, how do these patches get installed and patched? This gets a bit more complicated. The system utilizes OSTree to maintain a snapshot of the running partition and handle updates. Think of it as having two main drives that are initially a mirror of each other. On each boot, Silverblue with check trusted repositories for any patches or updates. It will download them and patch the non-running image causing the two copies to no longer be in sync. It switches the boot flag to the updated image causing it to be loaded on the next boot. Not only are you running the most current image at boot, but by having the second partition containing the old configuration, you can easily revert to it if one of the patches is not performing as expected.
Desktop immutability is the way to go for deploying standardized workstations while improving overall infrastructure security. For those wishing to apply the immutable philosophy to servers, I hope to have a write-up soon of how to accomplish this. The short version would be to rely on containerization such as Podman, Docker, LXD, or similar technology to run applications such as Apache in isolation. If someone gains access to a container, it is hard for them to modify the OS if there really isn't one.