Welcome to Linux4Science Discussions!

[sextonkennedy]

👋 Welcome!

We’re using Discussions as a place to connect with other members of our community. We hope that you:

• Ask questions you’re wondering about.
• Share ideas.
• Engage with other community members.
• Welcome others and are open-minded. Remember that this is a community we
  build together 💪.

To get started, comment below with an introduction of yourself and tell us about what you do with this community.

[sextonkennedy]

HI All,
I was an original member of the HSF startup team which later became the coordination team. Given that GitHub discussions is a new feature still in beta, I thought we could try it out for the purpose of gathering community input in a neutral 3rd party space that welcomes serious opinions about the future of Linux distributions in science.

[jouvin]

Good suggestion! HSF is also about promoting new ideas and tools!

[graeme-a-stewart]

Indeed - and I like the idea of conversations close to the code 👍

[alexiri]

Hi all! 👋 I'm Alex, from the CERN Linux team. I like the idea of using GitHub discussions!

[bonnieking]

Operating System Options for HEP

Hello, I'm part of the Scientific Linux group.

Here are some items that came out of dicussions at Fermilab, which we'd like to open up for input from the HSF community.

We should start with: What are our computing needs?

• Is a ten-year version lifespan a requirement? (Note that RHEL 3 and 4 were supported for seven years. Ten-year support started with RHEL/SL 5.)
• Can development practices/CI/CD improve our tolerance to OS changes?
• Do we require a zero-cost option?
• Would a no-cost but licensed distribution be an option (if Red Hat offers one)? Or do we require a product that can be installed without licensing?
• What are the risks/concerns with depending on an Upstream Vendor?

Other thoughts:

Red Hat’s universal base image may provide a stable platform for many/most uses, making the underlying platform less critical.

• This is, to some extent, free RHEL—same binaries, etc.—though additional packages have to be added from regular RHEL, CentOS, or another source.
• Underlying (host) platform is then less of an issue; can be Stream, Fedora, Ubuntu, Windows, or MacOS …

Possible options:

1. New RH free/reduced options

• https://www.redhat.com/en/blog/new-year-new-red-hat-enterprise-linux-programs-easier-ways-access-rhel
• Too limited for biggest use cases, but more options may be coming
• Easy to migrate; very close to what we have now
• Would have to track/get/renew licenses
• Extra effort for system administration would depend on details of distribution, etc.
• More options may be announced at a later time

2. Use CentOS Stream, as is

• Not a big change for users (smaller change than Modularity)
• 5-year lifespan is a big change; doesn’t fit LHC schedule as well
• No extra effort to distribute: no licenses, but more movement in packages
• Extra effort for system administration is hard to predict: More frequent upgrades—and the SL 6-to-7 upgrades took a lot of effort—but perhaps simpler upgrades. Not clear how much testing would be required after each upgrade.
• Quality has not been demonstrated

3. Effort to extend support for each Stream version

• An official SIG would have approved (and so far is explicitly not allowed)
• Years 6-10 should be pretty quiet; security fixes (and major bug fixes) only
• If it’s a shared effort, would need to work out who does what, how collaboration works (identity and authentication), etc.
• Effort for maintainers depends on our commitment, and on how the SIG works
• Where is the build system? (easier if it’s CentOS)
• Move to another RH clone: Springdale, Oracle, Rocky?
• Lots of communication/coordination issues with other HEP sites; again, containers will help some
• Springdale probably the best choice here; longer/better track record
• Rocky Linux is still being designed, though it appears to have a lot of initial support

5. Revive "Scientific Linux"

• Startup effort: 3-4 months (of 1 FTE); need new errata database, etc.
• Ongoing effort comparable to that for SL 6, 7

6. Move to Fedora

• Some advantages: newer kernel, better hardware support
• But very problematic from a stable-platform standpoint, for system administration, code validation, etc.
• Frequent updates would probably require containers for (nearly) everything

7. Move to a different Linux distribution: Debian, SuSE, …?

• Huge effort, both for support groups and for users. Probably unworkable unless all other options fail.
• Somewhat less for SuSE (though still a major change), which is RPM-based and generally more like RHEL

[brettviren]

@bonnieking a lot of good stuff there.

As I read it I think it is clear we must have this discussion in a framework where we identify the classes of computing and for each identify requirements along some lines (security, stability, support, hardware, users, software, ...) and then identify what constraints each class may impose on any of the others (eg, do interactive development servers require same OS as on scientific laptops).

Here is a low effort stab just to give a brief example of sort of what I think we need:

• Bulk computing (aka "grid", very large numbers of identical boxes, HPC/HTC may need proprietary drivers for some hardware, very limited and controlled access, some want years of total stability even forgoing security updates, expert maintained so licensed support not worth it)
• Servers on the Internet (few in number compared to bulk, web, email, ssh, uptime and security very important, buying into RHEL/RHEN may be useful here if not required as also tend to be expert-maintained)
• Servers inside institutional firewall (few in number, less uptime, less security requirements than above, access limited to LAN, more experimental applications, maybe less expert sysadmins compared to Internet facing)
• Interactive Scientific nodes (eg lxplus/gpvm/duneXXXX, many - mostly - trusted users, limited access, likely matching "bulk" platforms, stability but with more frequent security updates)
• Scientific workstation (order one user who tends also to be sysadmin, need specific software suites, tend to want latest version, lots of customization. developer environments, may require centralized user management)
• Scientific laptop (as above but add more physical accessibility security concerns, autonomous or no central user management)
• DAQ or other special purpose (maybe identify area that are so varied and special as to be explicitly outside of consideration)

As I consider "local optimization" in each class and the couplings between these classes I think part of a global optimum includes increasing our emphasis on portable software building with tools like Spack and/or augmenting a base OS with distributions like nix/guix.

I think I fully appreciate the concerns about effort of switching to Debian in point 7. They are real. But, if we could quantum-tunnel through that barrier I think we as a community would be in a much better well. I know it's a pipe dream!

[noggin143]

Along very similar lines, these were the initial thoughts presented to the WLCG management board in December - https://indico.cern.ch/event/958631/contributions/4150737/attachments/2162009/3648111/future%20linux%20v2.pdf

[DrDaveD]

(What happened to option 4?)

If it comes to a choice between RHEL clones, I'm certain that in a couple of months it's going to be obvious that Rocky is the choice. There is an enormous amount of contributed resources and inertia behind it and the Rocky Linux Foundation. I have an inside connection to the founder of Rocky (and co-founder of CentOS) and am following it fairly closely. It's going to be the clear category leader and do much better than CentOS ever did. Special Interest Groups are forming there. A Scientific Linux SIG there just like there was one for CentOS will make a lot of sense, I think. I don't think that reviving a complete Scientific Linux rebuild this year is going to make sense because there will be no way for it to compete. I think our community should be doing more to connect with the Rocky community if we want them to care about our needs.

[kpedro88]

I agree that the effort to migrate to non-RHEL-based Linux distribution should not be underestimated. However, my personal preference has always been Debian/Ubuntu, primarily because it's FOSS. While paid support is not as available compared to RHEL, free support is much more widely accessible. There's no worry that the package or support article that you need is behind a paywall (which can and does happen with RHEL). As physicists, we already rely heavily on the FOSS ecosystem to provide solutions to problems (since we often can't afford to pay for custom software). The recent advancements in container technology can be a significant help to preserve legacy workflows, though this isn't quite a panacea as mentioned below.

[stlammel]

We should not overlook that we are using quite a bit of third-party software in HEP. Not all are services that can go into a container or are software build by experiments.
It is not only support for the CentOS replacement of those add-ons but availability of them for each new OS release reduces the time span a release can be used actively. With migration planning toward the end, five year support can quickly result in a three year use.
CentOS was a nice consensus beyond HEP.

• Stephan

[brettviren]

This podcast may be useful to listen to. https://changelog.com/podcast/427

It does not erase my worry about basing our future computing ecosystem on capricious corporations (actually it reinforces it) but it does give me more understanding and confidence in the community building around Rocky Linux.

[gcooperfnal]

We plan to discuss these topics with experiments and other interested parties, at Fermilab and elsewhere in HEP. If you'd like to join, please email me (gcooper at fnal dot gov) and I'll send you Zoom information.

Friday, April 23, 2021
10:00–11:00 am US Central time