Next steps for HOMEBREW_INSTALL_FROM_API

[Rylan12]

Provide a detailed description of the proposed feature

Here are the tasks that need to be completed in order to consider HOMEBREW_INSTALL_FROM_API ready for official testing by maintainers and interested users:

• Improve brew update with HOMEBREW_INSTALL_FROM_API #13795: brew update should still update core and cask taps when HOMEBREW_INSTALL_FROM_API and HOMEBREW_DEVELOPER are both set (allowing maintainers to test this and still maintain homebrew/core and homebrew/cask)
• Allow developers to use dev commands and HOMEBREW_INSTALL_FROM_API #13815: Allow HOMEBREW_DEVELOPER users to use all dev commands even with HOMEBREW_INSTALL_FROM_API set
• Convert if MacOS.version blocks to on_system blocks homebrew-cask#137512: Migrate all casks to use on_{macos_version} blocks instead of if MacOS.version conditionals
• Remove the small number of Hardware::CPU methods that slipped through in formulae and casks
• Expand on_system rubocops #14301: Improve the rubocops to catch more cases
• Load casks from the JSON API with HOMEBREW_INSTALL_FROM_API #14304: Update CaskLoader to load casks from their JSON API like we do with formulae
• Include loaded_from_api key in Tab #13842: I think it might be a good idea to mark in the INSTALL_RECEIPT.json or .metadata directory whether the formula/cask was installed from the API in case we run into issues and it's helpful to know how a formula was installed. But, this is a lower priority and might not even be needed at all
• We need to handle download errors of the formula.json and cask.json files
• brew search fails when $HOMEBREW_INSTALL_FROM_API is set #14009
• Cannot install formulae on Linux when $HOMEBREW_INSTALL_FROM_API is set #13890
• We should have a way to be notified more reliably when formulae.brew.sh updates stop working as expected because this will mean that users can not get updates and will essentially be frozen in an old state. Maybe a bot should open an issue somewhere.
• brew.sh: test HOMEBREW_INSTALL_FROM_API on developers. #14384
• A warning or some sort of message should be shown when installing a formula with a post_install block or a cask with a {pre,post}flight block so that users know that some things may not work at the moment (or, in the case of formulae, they need to run brew postinstall <formula>)
  • See Add warning message to postinstall #14026 (comment) for a good solution
• HOMEBREW_INSTALL_FROM_API: make the default for everyone. #14412

What is the motivation for the feature?

I'm back at school so my availability over the next several months will be much more limited and less consistent. I'm opening this tracking issue to help monitor progress over time.

How will the feature be relevant to at least 90% of Homebrew users?

Eventually, HOMEBREW_INSTALL_FROM_API will be the default for everyone.

What alternatives to the feature have been considered?

None

[MikeMcQuaid]

[ ] A warning or some sort of message should be shown when installing a formula with a post_install block or a cask with a {pre,post}flight block so that users know that some things may not work at the moment (or, in the case of formulae, they need to run brew postinstall <formula>)

Is this something we expect to be fixed but just hasn't been yet? For a bottle installation, it should be able to use the post_install from within the bottle, right?

• I think it might be a good idea to mark in the INSTALL_RECEIPT.json or .metadata directory whether the formula/cask was installed from the API in case we run into issues and it's helpful to know how a formula was installed. But, this is a lower priority and might not even be needed at all

This is a good idea 👍🏻

I'm back at school so my availability over the next several months will be much more limited and less consistent. I'm opening this tracking issue to help monitor progress over time.

No worries, thanks for opening this tracking issue!

[Rylan12]

Is this something we expect to be fixed but just hasn't been yet? For a bottle installation, it should be able to use the post_install from within the bottle, right?

This requires us to build new bottles every time the postinstall method changes. And it's hard to tell whether existing bottles have outdated postinstall methods, so this might require rebottling everything. Also, style changes in the future may require rebottling all formulae that have postinstall blocks (currently 239 formulae) for the same reason. There's been some discussion in various places about the pros and cons of this as well as some other potential ways to deal with this, but I'll have to track those down

[MikeMcQuaid]

This requires us to build new bottles every time the postinstall method changes.

Yeh, I think this is fine for changes we need to push to users. Don't think this needs a warning as long as maintainers are made aware with docs/Slack communication.

And it's hard to tell whether existing bottles have outdated postinstall methods, so this might require rebottling everything.

This seems overkill.

Also, style changes in the future may require rebottling all formulae that have postinstall blocks (currently 239 formulae) for the same reason.

Why would a style change need rebottling?

but I'll have to track those down

That'd be great, thanks ❤️!

[Rylan12]

Why would a style change need rebottling?

I used the wrong words here. Most style changes that are truly only style changes should be fine as long as the old style isn't invalid ruby or anything like that. Really the issue is deprecations and stuff that, if not updated, might not be able to be understood by a brew version newer than the one that created the bottle.

but I'll have to track those down

We discussed this a bit on June 18 in Slack. I'll send a link over in slack to the messages. Here on GitHub, we discussed this in #12350, but there probably isn't anything in that discussion that didn't make it into the slack one.

[MikeMcQuaid]

Really the issue is deprecations and stuff that, if not updated, might not be able to be understood by a brew version newer than the one that created the bottle.

Gotcha 👍🏻

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[hyuraku]

@Rylan12

Migrate all casks to use on_{macos_version} blocks instead of if MacOS.version conditionals

I guess we need a new issue on https://github.com/Homebrew/homebrew-cask
to get the word out to people and have them modify the casks. How about it?

[p-linnane]

@hyuraku I am game to start working on these.

[Rylan12]

@hyuraku and @p-linnane, thanks for taking the initiative on this!

I think an issue in homebrew/cask would be great, but only once we've started the process and made sure all of its kinks are worked out (clearly from the past few days, we're not quite there yet 😅)

---

The immediate next steps that I think we need to take to get this ready are:

• Make brew readall simulate every OS/arch combination
• Fix homebrew/cask CI to not rely on checking the cask's contents for if MacOS.version conditionals

I plan on working on both of these soon, but if anyone else is interested feel free to give it a go

[hyuraku]

I think an issue in homebrew/cask would be great, but only once we've started the process and made sure all of its kinks are worked out (clearly from the past few days, we're not quite there yet 😅)

Understood, I will try to solve the other task.

[hyuraku]

@Rylan12
I have a question about We need to handle download errors of the formula.json and cask.json files
I couldn't find the code where we download cask.json; should we add the code to download cask.json by any chance?

[Rylan12]

We will once all casks are migrated to use on_monterey and friends. Only then (and maybe after some more fixes) will we be able to load casks from the JSON API. Right now, the existing cask API doesn't contain enough information to do so.

[apainintheneck]

I just realized that service blocks don't work with HOMEBREW_INSTALL_FROM_API just like the different *flight blocks. It'd be good to notify the user about that too.

[gromgit]

Another issue from way out in left field: https://github.com/orgs/Homebrew/discussions/4135

Cask install logs here: https://gist.github.com/gromgit/6a332888b9a77084cf40ae2e8c28d0c6

In a nutshell, the dotnet-sdk cask contains the following stanza:

  binary "/usr/local/share/dotnet/dotnet"

With HOMEBREW_INSTALL FROM API, what seems to be happening is Homebrew replaces /usr/local with $HOMEBREW_PREFIX, causing the symlinking to /opt/homebrew/bin on M1 Macs to fail. HOMEBREW_NO_INSTALL_FROM_API does not seem to do this, and therefore works correctly.

[carlocab]

Uninstalling casks when sudo is required seems to be broken with HOMEBREW_INSTALL_FROM_API. See Homebrew/homebrew-cask#140436.

[vincentisambart]

I have another probably that is very likely related to this: when brew upgrade builds code from source, I end up with an Empty installation error. Running it multiple times, or running brew update does not change anything. Adding a few prints here and there, some part of Homebrew is using an old version number of the formula, some part the most recent one. As the install path used and expected one do not match, the check if the installation is empty fails.

brew outdated tells me that my copy of some library is outdated, but looking at the local copy of the core tap it was not up to date (even if I run brew update manually). My guess is that due to this change towards using the API the copy of the tap is not updated. And when building from source, part of the Homebrew code uses the version from the API and part of it still uses the version number (or whole formula) from the local copy of the tap so that does not go well.

By the way running HOMEBREW_NO_INSTALL_FROM_API=1 brew upgrade did update the local copy and so fixed the problem so I guess I will have to rely on it until this problem is fixed.

[MikeMcQuaid]

when brew upgrade builds code from source

@vincentisambart Can you dump your brew config and brew doctor somewhere (even if there are warnings etc.) as we need to ensure that we never try to use the API when building from source.

[vincentisambart]

@MikeMcQuaid Today, a normal brew upgrade did upgrade some formulae that required building but this time everything went well... There might be an additional condition for the problem to appear (or some change in Homebrew itself might have fixed it). I still did a brew config and brew doctor: https://gist.github.com/vincentisambart/61284b0f4cd31ac7a04d5431a7ec1321

I should probably come back with brew config and brew doctor when the bug occurs again (if it does).

Update: Thinking more about it, it might have mainly happened not for patch version changes but only revision version changes (the ones that happen not when the package itself changed but when its dependencies do).

[MikeMcQuaid]

@vincentisambart Thanks. I may have fixed the underlying issue here.

[miccal]

Possibly related?

#14505