Update bottling Linux distribution to Ubuntu 22.04

[sjackman]

Provide a detailed description of the proposed feature

The current bottling distribution is Ubuntu 16.04. I propose updating the bottling distribution to a more recent version of Ubuntu.
See https://ubuntu.com/about/release-cycle

Distribution	Glibc	GCC
CentOS 7	2.17	4.8.5
Ubuntu 16.04	2.23	5.4.0
Ubuntu 18.04	2.27	7.5.0
Debian 10.12	2.28	8.3.0
Ubuntu 20.04	2.31	9.4.0
Debian 11.4	2.31	10.2.1
Ubuntu 22.04	2.35	11.2.0

What is the motivation for the feature?

Linux bottles ought to be built using the GCC provided by the bottling infrastructure. Some ~300 bottles fail to build using the GCC 5.4.0 provided by Ubuntu 16.04, because they require C++17 or C++20.

How will the feature be relevant to at least 90% of Homebrew users?

These ~300 bottles that currently depend on brewed GCC will no longer depend on brewed GCC.

What alternatives to the feature have been considered?

No good alternatives exist. It is widely accepted that we must upgrade the bottling infrastructure OS.

Tasks and open PRs

Please sort these tasks and PRs in the order that they ought to be completed and merged.

• audit_glibc: Permit glibc 2.27, 2.31, or 2.35 and fix the error message #13618
• gcc 12.1.0 homebrew-core#100411
• gcc 12.1.0 homebrew-core#106755
• Build bootstrap binaries for glibc on Linux glibc-bootstrap#3
• glibc 2.35 homebrew-core#106837
• Install the bottle dependencies #13751
• install glibc/gcc automatically if too old. #13577
• gcc_dep_if_needed: Install GCC if Glibc is too old #13758
• linux-headers: Change the default version to 5.15 homebrew-core#109272
• Automatically add symlinks from gcc cellar to HOMEBREW_PREFIX/lib #13770
• gcc@5: remove symlinks to lib homebrew-core#108816
• Update LINUX_CI_OS_VERSION to Ubuntu 22.04 #13733
• Tag Homebrew 3.6.0
• Use Ubuntu 22.04 to build bottles homebrew-core#108590
• Add messaging for homebrew/ubuntu16.04:master image deprecation #13819
• Remove Linux-only GCC dependencies homebrew-core#110010

[carlocab]

Ubuntu 18.04 is EOL next year, so I strongly advise against switching to that since I don't want us to be having this discussion again so soon.

Moving forward, I think it would be helpful to formalise and document the process of upgrading our bottling infrastructure, so it can be done more easily the next time we need to do it. Ideally, I'd like for it to be just changing some variables stored somewhere, but that might be asking for too much.

[sjackman]

Ubuntu 18.04 is EOL next year, so I strongly advise against switching to that since I don't want us to be having this discussion again so soon.

Note that LTS releases have security updates for ten years, and so Ubuntu 18.04 would continue to receive security updates until 2028.

For each Ubuntu LTS release, Canonical maintains the Base Packages and provides security updates, including kernel livepatching, for a period of ten years.

https://ubuntu.com/about/release-cycle

[carlocab]

I'm not as concerned about security updates as I am about being able to build new software. My impression is that a given Ubuntu version going into ESM is a nail in the coffin of that version's compiler: developers are less likely to consider it a target platform and are therefore more likely to adopt features that require a newer GCC.

That said, I won't oppose using Ubuntu 18.04 if we agree to not bottle any formulae that don't build with GCC 7.

To be clear, I think that we shouldn't bottle anything that doesn't build with the host compiler on Linux, regardless of the Ubuntu version. The intensity of this opinion is decreasing in the distro version we decide to use.

[MikeMcQuaid]

Thanks for opening @sjackman!

I propose updating the bottling distribution to a more recent version of Ubuntu.

@sjackman @danielnachun @iMichka can you spell out what you see as the pros and cons of e.g. Ubuntu vs. Debian vs. Centos? I'm interested in evaluations based on how up-to-date they are, how long they are supported for, etc.

Distribution

Might be worth adding to these a column for when they are supported/EOL/security updates only?

Linux bottles ought to be built using the GCC provided by the bottling infrastructure.

Does this mean "the default /usr/bin/gcc from the host operating system?

These ~300 bottles that currently depend on brewed GCC will no longer depend on brewed GCC.

What happens on OS versions that e.g. ship with an older GCC?

Please sort these tasks and PRs in the order that they ought to be completed and merged.

Some suggested additions from me:

• documentation about how to update Linux versions and a heuristic for deciding when to upgrade in future e.g. we upgrade to every Ubuntu LTS release within a year of its release
• the ability to use a matching host compiler on relevant systems where it's provided and install the relevant compiler elsewhere e.g. by doing brew install gcc@something automatically
  • bonus points if we somehow allow apt-get install gcc-something which is installed in /usr/bin/gcc-something and we still do the right thing

Moving forward, I think it would be helpful to formalise and document the process of upgrading our bottling infrastructure, so it can be done more easily the next time we need to do it. Ideally, I'd like for it to be just changing some variables stored somewhere, but that might be asking for too much.

I'm not as concerned about security updates as I am about being able to build new software. My impression is that a given Ubuntu version going into ESM is a nail in the coffin of that version's compiler: developers are less likely to consider it a target platform and are therefore more likely to adopt features that require a newer GCC.

Strongly agree with all of this. I'd like us to have a macOS-like process that involves us making small and regular updates to the host system we use.

To be clear, I think that we shouldn't bottle anything that doesn't build with the host compiler on Linux, regardless of the Ubuntu version. The intensity of this opinion is decreasing in the distro version we decide to use.

@carlocab i.e. never depends_on "gcc"? I would be cautiously in favour of this, partly as a forcing function for us to use a newer rather than older version of Ubuntu/Debian/CentOS.

[carlocab]

@carlocab i.e. never depends_on "gcc"?

Well, not never, but probably closer to it than where we are now. depends_on "gcc" is fine if you need fortran, but that's currently only a small share of the GCC dependents on Linux.

I think we need to grapple with the tradeoff between portability (i.e. working on older Linux distros) and bottle coverage. Trying to have them both results in Homebrew/core being too difficult to maintain, so I suggest we decide which one is more important to us.

[danielnachun]

Now that I've gotten glibc bootstrapping to working on CentOS 7 (see Homebrew/homebrew-core#106837), we no longer need to worry about portability to even the oldest support versions of Linux for at least the next several years, and I am hoping long before then that we'll have gotten glibc an :any cellar. glibc is the only place in brew or homebrew-core where we have to deal with concerns about portability - once it's installed all other bottles should work, and source builds should only fail due to opportunistic linkage or shim bypasses that we actively want to fix.

The only factor I now think we need to consider in how we decide on the upgrade schedule for CI is how many users we want to have to use brewed glibc and gcc. Given that gcc is relocatable and glibc should now very easy to build across a very broad range of systems, I am personally in favor of aggressively updating to the newest Ubuntu LTS as soon as it's available, meaning we would move to Ubuntu 22.04 LTS now.

While doing this will mean a larger percentage of our users will be relying on brewed gcc and glibc, the only major downside I can see to this is a modest increase in disk usage of ~350MB at most. There do seem to be occasional issues with build failures when using brewed glibc, but these always seem to be due to shim bypasses that we should actually try to fix. Furthermore, if the trend I observed while testing binary patching holds, about 95% of our non-relocatable formulae will become relocatable, making this mostly a moot point.

There may also be some failures even in CI when building older C++ packages with a newer GCC, but as one of the maintainers who has probably spent the most time dealing C++ API issues recently, I can say from personal experience that I would overwhelmingly prefer spending my time fixing older code to work with newer versions of GCC. These fixes usually entail nothing more than adding a few missing include statements, while the opposite problem of building with a newer GCC when the CI host is too old can be a total nightmare, especially if the formula has C++ dependents.

The advantage of more aggressive updating is that it makes maintenance much easier. If we're updating our compiler toolchain every 2 years to what is basically the latest version, this will mean our support for C++ standards on Linux will be on par with if not even better than what we have on macOS. In my view we should prioritize maintainer free time/sanity over saving users some disk space. However, I may be overlooking some other downsides to more aggressive updating and am open to hearing other opinions.

[iMichka]

@sjackman you might also want to add that we need to change the preferred gcc in https://github.com/Homebrew/brew/blob/9c03493774500cf16ced8938e1eb4eeae8216b20/Library/Homebrew/extend/os/linux/compilers.rb to your list?

[MikeMcQuaid]

Well, not never, but probably closer to it than where we are now. depends_on "gcc" is fine if you need fortran, but that's currently only a small share of the GCC dependents on Linux.

I think it might be worth us allowing depends_on "gfortran" to use an alias in this specific case to avoid confusion.

I think we need to grapple with the tradeoff between portability (i.e. working on older Linux distros) and bottle coverage. Trying to have them both results in Homebrew/core being too difficult to maintain, so I suggest we decide which one is more important to us.

Agreed. To me bottle coverage is unambiguously higher priority.

I am personally in favor of aggressively updating to the newest Ubuntu LTS as soon as it's available, meaning we would move to Ubuntu 22.04 LTS now.
...
The advantage of more aggressive updating is that it makes maintenance much easier. If we're updating our compiler toolchain every 2 years to what is basically the latest version, this will mean our support for C++ standards on Linux will be on par with if not even better than what we have on macOS. In my view we should prioritize maintainer free time/sanity over saving users some disk space.

Strongly agreed with all of this. I think we should aim to have similar C++ support on macOS and Linux. I don't mind if we have a longer build-from-source tail on both but we should be clearer with our installation and runtime messaging when we expect things may break and that people should submit PRs on Linux rather than filing issues (like we do with older macOS versions).

[danielnachun]

I think it might be worth us allowing depends_on "gfortran" to use an alias in this specific case to avoid confusion.

This would help to clarify things a lot and is very simple to implement.

Agreed. To me bottle coverage is unambiguously higher priority.

Fortunately there is no tension between portability and bottle coverage now that we better understand the glibc bootstrapping process. The real limitation in how old of a Linux system we can support is the build requirements for bootstrapping glibc, and these turn out to be far older than any system which is not EOL. I would not be surprised if our glibc 2.35 formula could even build on CentOS 6, which has been EOL for several years. The longest support cycle for any Linux distro is currently 13 years for SUSE Enterprise Linux but glibc could easily be bootstrapped on systems that are 15+ years old. So portability no longer even needs to factor into our considerations - glibc will work for all actively supported Linux systems.

Strongly agreed with all of this. I think we should aim to have similar C++ support on macOS and Linux. I don't mind if we have a longer build-from-source tail on both but we should be clearer with our installation and runtime messaging when we expect things may break and that people should submit PRs on Linux rather than filing issues (like we do with older macOS versions).

Once this migration is done I am going to work on finishing the CI part of binary prefix patching. Based on the testing I've already done, about 95% of non-relocatable formulae worked with no issues (and only some of the 5% that don't work even use C++), so I think the long tail will actually become very short, making even this concern mostly irrelevant. In the near future I expect that the only difference most Linux users will have if they are on an older system is the presence of the glibc and gcc@11 formulae.

[carlocab]

Fortunately there is no tension between portability and bottle coverage now that we better understand the glibc bootstrapping process.

Note: my earlier claim about the tradeoff between portability and bottle coverage is slightly more nuanced.

To rephrase it a bit: there is portability, bottle coverage, and maintainability of Homebrew and homebrew-core. Choose two.

The proposed solution for maintaining both portability and bottle coverage has introduced a lot of complexity into glibc (see Homebrew/homebrew-core#106837), so we're still giving up maintainability for portability and bottle coverage. That's what we've been doing so far, which is why I have been trying to nudge is toward the direction of prioritising maintainability more than we have been.

I've also opened a PR to try to ease the pain from this transition: #13631

[MikeMcQuaid]

The proposed solution for maintaining both portability and bottle coverage has introduced a lot of complexity into glibc (see Homebrew/homebrew-core#106837), so we're still giving up maintainability for portability and bottle coverage. That's what we've been doing so far, which is why I have been trying to nudge is toward the direction of prioritising maintainability more than we have been.

This is an important point 👍

[iMichka]

If the complexity is concentrated in the glibc formula, I'm happy with that. glibc is updated once every 6 months, and there is often no big rush to get it updated to the latest version. It's also: a linux-only formula / a formula that is not necessary if your Linux is new enough (which is the case for a lot of our users, looking at the stats provided by @sjackman).

[danielnachun]

Actually glibc only should be updated when the CI image is updated, so it would only get updated every 2 years if we follow the LTS release schedule, making this even less work to maintain.

[sjackman]

On 2022-07-29 I wrote…

The majority of our users are using Ubuntu 20.04 and Glibc 2.31. I suggest that we build bottles using Ubuntu 20.04 and Glibc 2.31. 79% of our users (23% on Glibc 2.35 and 56% on Glibc 2.31) will be able to use their host's Glibc and compiler without needing to installed the brewed Glibc and GCC.

If we use Ubuntu 22.04 and Glibc 2.35 to build bottles, only 23% of our users will have a new enough version of Glibc on their host system to use our bottles, and the rest will need to install the brewed glibc and gcc. A system with a single version of Glibc is less complex and easier to administer than a system with multiple versions of Glibc, which is pretty unusual. I would prefer that a majority of our users be able to use their own host's Glibc and compiler, as we do on macOS.

That said, I'm open to using either Ubuntu 20.04 or Ubuntu 22.04 for building bottles, whichever we settle on. I'd be curious to know how many bottles cannot be built using the compiler provided by Ubuntu 20.04 (GCC 9.4.0), but could be built by Ubuntu 22.04 (GCC 11.2.0).

#13625 (comment)
@MikeMcQuaid wrote…

I'd probably like to see something like e.g. we use the latest LTS at least ~3-4 months after release and we always migrate to the latest LTS within at most 12 months of release.

This timeline seems reasonable to me.

[danielnachun]

A system with a single version of Glibc is less complex and easier to administer than a system with multiple versions of Glibc, which is pretty unusual.

Can you elaborate on this? I don't want to totally dismiss this concern but I think it would be great to have more concrete examples of how this a problem. Speaking from personal experience, the only times I've seen problems with brewed glibc in my system have been when brewed and non-brewed libraries/binaries get mixed up i.e. linking against a brewed library while using a non-brewed GCC that ends up using the older host glibc. While these problems are unfortunate, there are some pretty serious downsides to using an older GCC to build formulae that may outweigh these concerns

I'd be curious to know how many bottles cannot be built using the compiler provided by Ubuntu 20.04 (GCC 9.4.0), but could be built by Ubuntu 22.04 (GCC 11.2.0).

The even harder question to answer here is: how many bottles will not be able to built with GCC 9 but can be built with GCC 11 in 2 years when we would be preparing to transition to 22.04? It's very difficult to predict this but my concern is that if it gets too large, we will be stuck back at the same issue we've struggled with before. While I strongly oppose completely blocking the use of brewed GCC for any formulae in Linux, I think we should do everything we can to keep the number of formulae that need this as small as possible to maximize maintainability.

[carlocab]

To be clear: I agree that these symlinks need to go.

Let's track this in an issue somewhere.

#13756

[sjackman]

I'd like to have a stable release of Homebrew that migrates our users to glibc 2.35 and gcc@11 before we move our bottling infrastructure to Ubuntu 22.04.

I'd like to move the bottling infrastructure at the same time we release 3.6.0.

I should clarify. What I mean is that the precise timing of how the PRs are merged is important.
The PRs that bump the user's version of Glibc to 2.35 and GCC to 11 must be merged and a stable release of Homebrew tagged before we start building bottles that use those versions. It can all happen in quick succession, but it must happen in that order.

[MikeMcQuaid]

I should clarify. What I mean is that the precise timing of how the PRs are merged is important.
The PRs that bump the user's version of Glibc to 2.35 and GCC to 11 must be merged and a stable release of Homebrew tagged before we start building bottles that use those versions. It can all happen in quick succession, but it must happen in that order.

Yup, good with me 👍🏻

What I'd suggest is that we have a single "Homebrew on Linux 3.6.0" homebrew-core PR that we can get green and merge at exactly the same time as we tag. How''s that sound?

[danielnachun]

As far as I know, all blocking PRs have been merged so once we've settled the issues in #13733 and Homebrew/homebrew-core#108590 we can ship them!

[carlocab]

I'd want to work out what is happening here first, as it's an early trial of the new glibc: Homebrew/discussions#3601

[danielnachun]

I'd want to work out what is happening here first, as it's an early trial of the new glibc: Homebrew/discussions#3601

I gave a much longer answer there but the TLDR is that it was caused by mixing brewed and system libraries when brewed glibc is installed. This can cause breakage and should be considered an unsupported use case. It's not a new problem but since more uses will be use glibc now we'll just have to make sure it's documented properly.

[carlocab]

While the process is fresh in your minds -- could someone write up a guide for updating the bottling distribution, so we're not making it up as we go along next time?

[danielnachun]

I am happy to work on a guide. Where should it be added?

[carlocab]

Could be added as a maintainer guide at https://github.com/Homebrew/brew/tree/master/docs#maintainers.

[sjackman]

or https://github.com/Homebrew/brew/blob/master/docs/Linux-CI.md ?

[carlocab]

or https://github.com/Homebrew/brew/blob/master/docs/Linux-CI.md ?

That's user-facing documentation. They don't need to see the details of how we update the distro there (but they can read it in the other docs if they're interested).

[MikeMcQuaid]

or https://github.com/Homebrew/brew/blob/master/docs/Linux-CI.md ?

This would make sense to me as it already references e.g. versions.

[MikeMcQuaid]

But anywhere in docs.brew.sh works for me 👍🏻

[danielnachun]

All Linux-only GCC dependencies have been removed from homebrew-core, so I think we can officially declare the migration complete!

[MikeMcQuaid]

Great work, thanks @danielnachun and everyone!