Automate bootstrapping of curl and git on Linux

[danielnachun]

Provide a detailed description of the proposed feature

We have completed our transition to a newer CI image on Linux and I wanted to address the other Linux-related subject I believe needs some refinement: the bootstrapping process before we can install any formulae. Currently, we require curl 7.41.0 and git 2.7.0 to be installed before any formulae can be installed. This can complicate things if the host OS doesn’t offer newer versions of these and git is often not installed in Docker images by default either. It has been brought up as an issue regularly:

• Error: Please update your system cURL install#367
• Why does Brew require a newer version of curl?? discussions#224
• brew not using newer curl (or git) in PATH, complains that it is an old version #13071
• https://stackoverflow.com/questions/70300091/installing-brew-on-linux-without-sudo-and-older-version-git-and-curl

The goal here would be very simple – find an automated way for Homebrew to bootstrap itself enough on Linux to the point where brewed curl and git can be installed. There is no intention here to change anything about Homebrew functions once those two formulae are installed, and anything which might be used temporarily in the bootstrapping can be discarded afterward.

I have a few possible solutions that I came up with based on the follow criteria:

1. It cannot require any changes to homebrew-core. We want to separate the maintenance of formulae from the maintenance of installation routines.
2. It must be easily testable in CI.
3. It doesn’t require using the host compiler toolchain to install anything from source - building Homebrew formulae from source with the Homebrew toolchain is totally fine. My reason for wanting to avoid the host toolchain is that it will be a lot more heterogeneous than using Homebrew’s toolchain and therefore more prone to failure.

Solutions for curl:

1. Use a portable-curl similar to portable-ruby to avoid the need for an outside curl. It would be bult in its own repo modeled after portable-ruby and would be updated automatically with livecheck. A portable-git is not possible because it’s not relocatable.

2. Add a DownloadStrategy that uses the HTTP gem from Ruby while installing curl and its dependencies - git can be installed afterward with brewed curl. This is similar to how Anaconda downloads its packages with Python. This would only be needed to install curl so that the CurlDownloadStrategy would work, and would have an appropriate unit test.

Solutions for git:

1. Download a tar.gz release of brew and use the GitHub API to download a copy of homebrew-core locally instead of cloning it with git.

Solutions for both curl and git:

1. Use a miniconda installation to install curl and git binaries in a conda environment.

What is the motivation for the feature?

It will make Homebrew much easier to install on Linux systems without sudo, on a host system that does provide new enough curl or git packages or in more minimal Docker containers.

How will the feature be relevant to at least 90% of Homebrew users?

It will be relevant to any potential Linux user whose host system's curl is too old or whose git is too old or missing. It does not affect existing users.

What alternatives to the feature have been considered?

The status quo.

[MikeMcQuaid]

1. Use a portable-curl similar to portable-ruby to avoid the need for an outside curl. It would be bult in its own repo modeled after portable-ruby and would be updated automatically with livecheck. A portable-git is not possible because it’s not relocatable.

We should not do this. It will need updated too often.

2. Add a DownloadStrategy that uses the HTTP gem from Ruby while installing curl and its dependencies - git can be installed afterward with brewed curl. This is similar to how Anaconda downloads its packages with Python. This would only be needed to install curl so that the CurlDownloadStrategy would work, and would have an appropriate unit test.

This is a good idea.

Another 3. option: use the system curl regardless of the version to bootstrap a curl formula and let the user sort out any issues that result from this.

1. Download a tar.gz release of brew and use the GitHub API to download a copy of homebrew-core locally instead of cloning it with git.

We used to do this but its a bit messy. Instead I think we should use and rely on HOMEBREW_INSTALL_FROM_API.

on a host system that does provide new enough curl or git packages or in more minimal Docker containers.

Can we use our analytics to estimate how many of these systems there are?

[carlocab]

1. Use a portable-curl similar to portable-ruby to avoid the need for an outside curl. It would be bult in its own repo modeled after portable-ruby and would be updated automatically with livecheck. A portable-git is not possible because it’s not relocatable.

We should not do this. It will need updated too often.

Agreed.

Another 3. option: use the system curl regardless of the version to bootstrap a curl formula and let the user sort out any issues that result from this.

Alternatively: improve documentation for providing your own source-built curl that was installed without sudo using HOMEBREW_CURL_PATH.

We can print links to this documentation whenever brew complains about /usr/bin/curl being too old. If desired, we can add a test in CI that does

HOMEBREW_CURL_PATH="$HOME/bin/curl" brew fetch foo

to make sure that this works, but I'm ambivalent about this.

We used to do this but its a bit messy. Instead I think we should use and rely on HOMEBREW_INSTALL_FROM_API.

Agreed.

It will make Homebrew much easier to install on Linux systems without sudo, on a host system that does provide new enough curl or git packages or in more minimal Docker containers.

I'd rather not make it easier to install Homebrew without any use of sudo at all.

If we make this too easy, then users who might've gone out of their way to use sudo for the install script once (but then never again after that) are more likely to just use the sudo-less route but end up without having access to non-relocatable bottles.

Making the sudo-less route a little painful makes it more likely that users who can install in the default prefix will, and that leads to fewer problems for both them and us.

I am willing to change my position on this if we're able to make all important formulae (e.g. python, git) relocatable.

Regarding minimal Docker containers: to me, it would be much better to provide a minimal Docker container that has brew pre-installed (perhaps even from a Homebrew/brew tarball, to minimise size) and uses HOMEBREW_INSTALL_FROM_API.

I also agree with Mike regarding analytics about systems with curl or git that are too old.

[danielnachun]

2. Add a DownloadStrategy that uses the HTTP gem from Ruby while installing curl and its dependencies - git can be installed afterward with brewed curl. This is similar to how Anaconda downloads its packages with Python. This would only be needed to install curl so that the CurlDownloadStrategy would work, and would have an appropriate unit test.

This is a good idea.

I will work on this as soon as I can, as it is relatively straight forward.

1. Download a tar.gz release of brew and use the GitHub API to download a copy of homebrew-core locally instead of cloning it with git.

We used to do this but its a bit messy. Instead I think we should use and rely on HOMEBREW_INSTALL_FROM_API.

What's "messy" about this? I'd love to be able to use HOMEBREW_INSTALL_FROM_API but it doesn't currently work for non-relocatable packages or packages that have postinstall blocks, so it doesn't quite get us there. I'd rather spend my limited time at the moment fixing up whatever issues there are with bootstrapping without git because that does work but seems to need polishing. I will implement whatever testing is needed to make sure it's well behaved.

Can we use our analytics to estimate how many of these systems there are?

I don't think we currently collect this level of detail about our hosts, but it might be useful to add at some point.

[carlocab]

I'd rather spend my limited time at the moment fixing up whatever issues there are with bootstrapping without git because that does work but seems to need polishing.

We should focus on issues with glibc being keg-only first. Additionally, if we did want to fix bootstrapping without Git, we should focus on doing this for installs in the default prefix.

Can we use our analytics to estimate how many of these systems there are?

I don't think we currently collect this level of detail about our hosts, but it might be useful to add at some point.

There is some distro information in our analytics. See #13619 (comment). Can we infer or estimate numbers from here?

[MikeMcQuaid]

I'd rather spend my limited time at the moment fixing up whatever issues there are with bootstrapping without git because that does work but seems to need polishing.

We should focus on issues with glibc being keg-only first. Additionally, if we did want to fix bootstrapping without Git, we should focus on doing this for installs in the default prefix.

I strongly agree. I'm not going to ✅ any PRs related to this functionality in this issue until we've had at least a week of stability with the Ubuntu 22.04/glibc/gcc transition. At the time of writing here: we have CI failing all over the Homebrew organisation due to this transition.

Can we use our analytics to estimate how many of these systems there are?

I don't think we currently collect this level of detail about our hosts, but it might be useful to add at some point.

There is some distro information in our analytics. See #13619 (comment). Can we infer or estimate numbers from here?

Yes, please. This should be done before writing code and prioritising work, not after.

1. Download a tar.gz release of brew and use the GitHub API to download a copy of homebrew-core locally instead of cloning it with git.

We used to do this but its a bit messy. Instead I think we should use and rely on HOMEBREW_INSTALL_FROM_API.

What's "messy" about this? I'd love to be able to use HOMEBREW_INSTALL_FROM_API but it doesn't currently work for non-relocatable packages or packages that have postinstall blocks, so it doesn't quite get us there.

It's messy because:

• we download a tarball from a Git repository which does have a .git which we'll need to initialise later (from Git) anyway
• we have HOMEBREW_INSTALL_FROM_API for this case
• it does work for post_install blocks, it just uses them from the bottle
• it doesn't work for non-bottles which aren't a supported workflow anyway

I don't want additional work going into improving unsupported workflows when supported workflows are so broken today.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.