Abandoning Scientific Linux Was a Mistake

A recent CERN announcement caught my attention. By the end of 2026, more than 2,200 industrial computers and embedded systems around CERN's accelerator complex are expected to be running Debian 13.

CERN is not abandoning the Red Hat ecosystem. AlmaLinux and RHEL remain the main supported Linux distributions across much of the organization; Debian support is currently limited to accelerator front-end systems.

Still, the move took me back to a decision made more than a decade ago: CERN and Fermilab's gradual abandonment of Scientific Linux. With hindsight, I think it was a mistake.

Not because Scientific Linux was technically superior to CentOS, or because maintaining another Linux distribution was free. And not because CERN or Fermilab could somehow have controlled what Red Hat or IBM later chose to do.

The mistake was treating Scientific Linux mainly as duplicated engineering work. It was infrastructure.

What Scientific Linux was solving

Scientific Linux grew out of a practical problem in High Energy Physics. Large experiments span laboratories, universities, computing centers, and countries. If one site builds against one version of glibc, another uses something slightly different, and a third runs an entirely different packaging environment, things become painful very quickly.

The scientific community also needed an unusual combination: a Linux distribution that would remain stable for years, work with enterprise software, be freely redistributable, and run across institutions without requiring a commercial license for every machine.

Red Hat Enterprise Linux provided the stability and long lifecycle, and Red Hat published the source needed to rebuild it. Scientific Linux turned that source into a community resource. Fermilab announced the distribution in 2003, and CERN joined soon afterward. It eventually spread far beyond those two laboratories. Universities, research institutions, experiments, companies, and even systems aboard the International Space Station used it or distributions derived from it.

Scientific Linux was never simply "RHEL with a different wallpaper." It gave the scientific community an institutionally independent implementation of the Enterprise Linux platform.

Then CentOS looked like the obvious answer

When Red Hat and CentOS joined forces in 2014, moving to CentOS looked perfectly reasonable. CentOS already offered what many Scientific Linux users wanted: a freely available Enterprise Linux rebuild backed by a much larger general-purpose community.

CERN began moving its next major release from Scientific Linux CERN to CERN CentOS 7. Scientific Linux 5 and 6 remained supported, but the future platform at CERN would be based on CentOS. The rationale was compelling. Why should CERN and Fermilab spend scarce engineering time rebuilding a Linux distribution when CentOS was already doing essentially the same work?

Why maintain an HEP-specific distribution when the scientific community could converge on a larger common platform? If Red Hat itself was supporting the CentOS project, that seemed to make the platform more sustainable, not less. In 2019, Fermilab took the argument to its logical conclusion—or its literal "end." There would be no Scientific Linux 8. Fermilab would deploy CentOS 8 instead and work with CERN and other laboratories to improve CentOS for high-energy physics.

Taken on its own, this was not an irrational decision. But it removed something that was hard to see on a spreadsheet: a credible exit.

A distribution has option value

This is what I think we underestimated.

The cost of maintaining Scientific Linux was visible. People had to rebuild packages, test updates, maintain repositories, produce releases, handle security updates, and support users.

The benefit of an independent distribution was harder to quantify. As long as CentOS behaved exactly as CERN and Fermilab expected, Scientific Linux looked redundant. That is how redundancy always looks when nothing has failed yet.

Scientific Linux gave the scientific community its own implementation of a RHEL-compatible computing environment. More importantly, it kept alive the people, processes, infrastructure, and institutional knowledge needed to maintain one.

That capability had option value. You might not need to exercise the option this year, or even this decade. It becomes valuable when the assumptions underneath your primary platform change. Those assumptions changed surprisingly quickly.

[!NOTE] CERN uses a similar argument to justify the FCC: the knowledge required to build large machines such as accelerators and detectors is valuable and must be preserved. If Europe does not build the next machine after the LHC, that expertise may be lost, the next large machine may be built in China, and Europe may lose its technological leadership in the field. The same argument applies to software infrastructure.

The CentOS assumption did not last

In December 2020, Red Hat changed the role of CentOS.

CentOS Linux—the traditional downstream rebuild of released RHEL versions—was discontinued in favor of CentOS Stream, which sits ahead of RHEL rather than behind it. This was more than a change in release cadence.

Organizations had standardized on CentOS because they wanted a freely distributable approximation of the current RHEL release. The product they had standardized on effectively ceased to exist.

CERN and Fermilab immediately had to reconsider their Linux strategy. CERN noted that the shorter CentOS Stream lifecycle was incompatible with some scientific use cases. The laboratories evaluated Stream, RHEL licensing arrangements, and the emerging Enterprise Linux rebuilds. By 2022, CERN and Fermilab were recommending AlmaLinux as the standard distribution for experiments.

The circularity is hard to miss.

Scientific Linux had been retired partly because maintaining another RHEL rebuild seemed unnecessary while CentOS existed. A few years later, the community needed an independent RHEL-compatible distribution again, so CERN and Fermilab adopted another independently governed RHEL-compatible distribution.

AlmaLinux is a good project. This is not a criticism of it. The requirement never disappeared; only our implementation of it did.

Then the ground moved again

In 2023, Red Hat changed how RHEL-related sources were publicly distributed. CentOS Stream became the sole public repository for that source material, replacing the previous git.centos.org publication model.

This did not make RHEL "closed source," despite how the change was sometimes described. That characterization would be inaccurate. But it made the dependency structure of the Enterprise Linux rebuild ecosystem much more obvious.

By then, the lesson should have been familiar: technical compatibility with an upstream product is not the same as independence from the organization that controls it.

Scientific Linux had strategic value that was never properly accounted for. Its existence meant CERN, Fermilab, and other institutions were not merely consumers of an ecosystem. Together, they could reproduce a critical part of that ecosystem themselves.

Once discarded, that capability is much harder to recreate than to keep alive.

And now CERN is moving part of the accelerator complex to Debian

The latest chapter makes this history particularly interesting. CERN's accelerator controls group first tried to remain within the Red Hat ecosystem. According to the presentation at MiniDebConf Winterthur 2026, the original plan involved CentOS Stream, with Debian prepared as a fallback because, in the presenters' words, they could not afford new surprises.

Then came a more physical problem. RHEL 9 raised its x86-64 baseline to x86-64-v2. RHEL 10 moved to x86-64-v3.

Replacing old hardware may be inconvenient but manageable for an ordinary server fleet. Accelerator controls are not an ordinary server fleet.

These computers interface with custom electronics, specialized boards, legacy buses, real-time systems, and equipment installed throughout a vast accelerator complex. The hardware can remain operational for decades because replacing one computer may mean redesigning the electronics connected to it.

CERN's 2023 risk analysis estimated that staying in the Red Hat ecosystem could cost about 5.4 million CHF. It could also require redesigning around eleven boards, hiring more engineers and technicians, reorganizing racks, recabling systems, and recommissioning equipment.

The solution was refreshingly simple: do not replace millions of francs' worth of functioning hardware to satisfy an operating system's CPU baseline. Replace the operating system.

CERN chose Debian. By the end of 2026, more than 2,200 industrial computers and embedded systems around the accelerator complex are expected to run Debian 13.

The irony is difficult to miss

What interests me most is not that CERN chose Debian, but why Debian was valuable.

The accelerator team specifically praised its continued support for older and less common architectures and its ability, as a community-led distribution, to compete with Red Hat.

CERN is also sponsoring Freexian to strengthen Debian's long-term-support ecosystem. There is an important principle here: when an institution depends on infrastructure supplied by a community project, contributing resources to keep that project independent and healthy may cost less than consolidating everything around one vendor ecosystem in pursuit of short-term efficiency.

That sounds remarkably close to what Scientific Linux used to provide.

Scientific Linux was maintained by a surprisingly small group

This is another reason I question whether ending Scientific Linux produced the savings we assumed. The distribution was not maintained by hundreds of engineers.

Its project history lists only a handful of major developers in its later years, with Fermilab providing the main sponsorship, build infrastructure, bandwidth, and website. That does not make the work trivial. Release engineering, security updates, testing, package rebuilds, infrastructure, and user support all take real time.

But those costs should be compared with the value they produced. Scientific Linux provided a stable computing platform to an enormous scientific ecosystem for roughly two decades. Preserving that capability should not have been evaluated solely in terms of the engineer-hours saved by adopting CentOS.

The relevant question was: what would it cost to lose the ability to operate independently if the surrounding ecosystem changed?

The past decade gives us at least a partial answer.

Did Scientific Linux keep Red Hat honest?

There is a tempting, stronger argument: that Scientific Linux and other independent rebuilds constrained Red Hat, and that removing one of the major alternatives helped concentrate power around CentOS and RHEL.

I think there is something to it, but I would not present it as historical fact. We cannot know the counterfactual.

Scientific Linux probably would not have prevented Red Hat from changing CentOS. IBM completed its acquisition of Red Hat in July 2019, after Fermilab had already announced that there would be no Scientific Linux 8. It would therefore be misleading to reduce this to a story about IBM killing something CERN and Fermilab should have anticipated.

What I do believe is that independent alternatives change incentives. A vendor behaves differently when major customers and institutions have a credible exit than when leaving requires them to rebuild years of infrastructure.

This is not specific to Red Hat. It is basic dependency management. Competition matters even when nobody switches; the possibility of switching matters too. Scientific Linux provided that possibility.

And because major scientific institutions maintained it rather than a commercial Linux vendor, its incentives were unusually well aligned with long-lived scientific infrastructure.

The real mistake was confusing standardization with dependency

Standardization is good.

Particle physics could not operate efficiently if every university and laboratory invented its own incompatible computing platform. CERN and Fermilab were right to want common interfaces, compatible binaries, common packaging, and predictable operating environments. But standardization and monoculture are not the same thing.

The strategy went wrong when we assumed that standardization required abandoning an independent implementation of the standard. Scientific Linux could have remained boring.

In fact, boring was exactly what we needed from it. It did not need to compete with Fedora on innovation or Ubuntu on desktop adoption. It needed to remain a reproducible, institutionally controlled Enterprise Linux platform: somewhere the scientific community could go if the commercial ecosystem moved in a direction incompatible with scientific computing.

That is valuable infrastructure even when almost nobody notices it. Especially then.

The lesson is larger than Scientific Linux

I do not think the answer today is to resurrect Scientific Linux. The ecosystem has moved on. AlmaLinux exists. Rocky Linux exists. Debian is proving increasingly useful at CERN. Containers and modern software distribution have reduced the importance of the host operating system for many workloads.

The lesson I take from Scientific Linux is about how research institutions value infrastructure. We are good at calculating the immediate cost of maintaining something ourselves. We are much worse at calculating the long-term cost of losing the ability to maintain it.

When deciding whether to retire institutional open-source infrastructure, the calculation should include more than maintenance hours. It should account for governance, concentration risk, the cost of migration if an upstream project changes direction, the institutional knowledge being discarded, and the value of a credible alternative that you hope never to need.

The CERN accelerator team's final advice in its Debian presentation is probably the best summary:

"Distribution portability is a very good thing."

It took more than a decade, several Enterprise Linux strategy changes, a new community rebuild, and now the migration of thousands of accelerator computers to arrive there. Scientific Linux was already teaching us that lesson twenty years ago.

We just stopped listening.