HOMEBREW_ARTIFACT_DOMAIN behaviour differs from documentation

[UiP9AV6Y]

brew config output

HOMEBREW_VERSION: 3.4.9
ORIGIN: https://github.com/Homebrew/brew
HEAD: d0a0bbef8db45cca303c607037338d02d8eaa8ca
Last commit: 5 days ago
Core tap ORIGIN: https://github.com/Homebrew/homebrew-core
Core tap HEAD: d253f6be78f6352f300e1e7070b65217e1c17256
Core tap last commit: 4 hours ago
Core tap branch: master
HOMEBREW_PREFIX: /opt/homebrew
HOMEBREW_ARTIFACT_DOMAIN: https://cache.my.corp/docker-github
HOMEBREW_CASK_OPTS: []
HOMEBREW_CORE_GIT_REMOTE: https://github.com/Homebrew/homebrew-core
HOMEBREW_DOCKER_REGISTRY_BASIC_AUTH_TOKEN: set
HOMEBREW_MAKE_JOBS: 2
HOMEBREW_NO_ANALYTICS: set
Homebrew Ruby: 2.6.8 => /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/bin/ruby
CPU: dual-core 64-bit dunno
Clang: 13.1.6 build 1316
Git: 2.32.0 => /Library/Developer/CommandLineTools/usr/bin/git
Curl: 7.79.1 => /usr/bin/curl
macOS: 12.3.1-arm64
CLT: 13.3.0.0.1.1645755326
Xcode: N/A
Rosetta 2: false

brew doctor output

Your system is ready to brew.

Verification

• I ran brew update and am still able to reproduce my issue.
• I have resolved all warnings from brew doctor and that did not fix my problem.

What were you trying to do (and why)?

we use brew in our CI system and want to avoid hitting upstream servers with too many requests. therefor we want to utilize a cache.

What happened (include all command output)?

installing casks/formulae with HOMEBREW_ARTIFACT_DOMAIN configured
results in URLs being rewritten in two different ways.

brew install wget

==> Downloading https://ghcr.io/v2/homebrew/core/gettext/manifests/0.21
==> Downloading from https://cache.my.corp/docker-github/v2/homebrew/core/gettext/manifests/0.21
==> Downloading https://ghcr.io/v2/homebrew/core/gettext/blobs/sha256:6e2c829031949c0cbd758d0701ed62c191387736e76a98a046c0619907632225
==> Downloading from https://cache.my.corp/docker-github/v2/homebrew/core/gettext/blobs/sha256:6e2c829031949c0cbd758d0701ed62c191387736e76a98a046c0619907632225
==> Downloading https://ghcr.io/v2/homebrew/core/libunistring/manifests/1.0
==> Downloading from https://cache.my.corp/docker-github/v2/homebrew/core/libunistring/manifests/1.0
==> Downloading https://ghcr.io/v2/homebrew/core/libunistring/blobs/sha256:b8b2f6fe30eefd002bf0dbb5fc0e5c6dc0d5f9b9219f4d6fcddc48e3bc229b23
==> Downloading from https://cache.my.corp/docker-github/v2/homebrew/core/libunistring/blobs/sha256:b8b2f6fe30eefd002bf0dbb5fc0e5c6dc0d5f9b9219f4d6fcddc48e3bc229b23
==> Downloading https://ghcr.io/v2/homebrew/core/libidn2/manifests/2.3.2
==> Downloading from https://cache.my.corp/docker-github/v2/homebrew/core/libidn2/manifests/2.3.2
==> Downloading https://ghcr.io/v2/homebrew/core/libidn2/blobs/sha256:fcc51d2c385b19d647da5a53f7041f13f97d2c1119a7cbbfd8433e9e55bf5012
==> Downloading from https://cache.my.corp/docker-github/v2/homebrew/core/libidn2/blobs/sha256:fcc51d2c385b19d647da5a53f7041f13f97d2c1119a7cbbfd8433e9e55bf5012
==> Downloading https://ghcr.io/v2/homebrew/core/ca-certificates/manifests/2022-04-26
==> Downloading from https://cache.my.corp/docker-github/v2/homebrew/core/ca-certificates/manifests/2022-04-26
==> Downloading https://ghcr.io/v2/homebrew/core/ca-certificates/blobs/sha256:c05a44feba2a630de2e1cefba90d3aa3f74e4d57146c0117858f648c419abeae
==> Downloading from https://cache.my.corp/docker-github/v2/homebrew/core/ca-certificates/blobs/sha256:c05a44feba2a630de2e1cefba90d3aa3f74e4d57146c0117858f648c419abeae
==> Downloading https://ghcr.io/v2/homebrew/core/openssl/1.1/manifests/1.1.1n
==> Downloading from https://cache.my.corp/docker-github/v2/homebrew/core/openssl/1.1/manifests/1.1.1n
==> Downloading https://ghcr.io/v2/homebrew/core/openssl/1.1/blobs/sha256:f9dbd826c0a48bb37a7bd182f3bbcad0376b22a72c4a0b0d152c9e40bfb716b2
==> Downloading from https://cache.my.corp/docker-github/v2/homebrew/core/openssl/1.1/blobs/sha256:f9dbd826c0a48bb37a7bd182f3bbcad0376b22a72c4a0b0d152c9e40bfb716b2
==> Downloading https://ghcr.io/v2/homebrew/core/wget/manifests/1.21.3
==> Downloading from https://cache.my.corp/docker-github/v2/homebrew/core/wget/manifests/1.21.3
==> Downloading https://ghcr.io/v2/homebrew/core/wget/blobs/sha256:fc83eec77acee50d2d7ce3bb0cca08d80acccc148e909921de42e57dd5fc7f3d
==> Downloading from https://cache.my.corp/docker-github/v2/homebrew/core/wget/blobs/sha256:fc83eec77acee50d2d7ce3bb0cca08d80acccc148e909921de42e57dd5fc7f3d

brew install --casks corretto

==> Downloading https://corretto.aws/downloads/resources/18.0.1.10.1/amazon-corretto-18.0.1.10.1-macosx-aarch64.pkg
==> Downloading from https://cache.my.corp/docker-github/https://corretto.aws/downloads/resources/18.0.1.10.1/amazon-corretto-18.0.1.10.1-macosx-aarch64.pkg
curl: (22) The requested URL returned error: 401
Error: Download failed on Cask 'corretto' with message: Download failed: https://corretto.aws/downloads/resources/18.0.1.10.1/amazon-corretto-18.0.1.10.1-macosx-aarch64.pkg

What did you expect to happen?

asset URLs hosted under ghcr.io get transformed into a format understood by container registry mirrors such as harbor and artifactory. this is also the documented behaviour.

all other URLs get simply prefixed with the configured artifact domain, resulting in a protocol scheme to appear twice in the URL (https://cache.my.corp/docker-github/https://example.com/foo.tar.gz). i am not sure, which software is supposed to handle URLs formatted like this.

in another issue it is explained, that HOMEBREW_ARTIFACT_DOMAIN is inteded to be used with proxies, which i do not understand, given that http_proxy et al. are the environment variables to be used with HTTP proxies.

given the lack of unit tests for this feature, i do not know if the observed behaviour is correct and the documentation is incomplete or vice-versa.

Step-by-step reproduction instructions (by running brew commands)

`env HOMEBREW_ARTIFACT_DOMAIN=https://cache.my.corp/docker-github \
brew install --casks corretto`
`env HOMEBREW_ARTIFACT_DOMAIN=https://cache.my.corp/docker-github \
brew install wget`

[MikeMcQuaid]

i do not know if the observed behaviour is correct and the documentation is incomplete

Observed behaviour is correct and documentation is incomplete.
Could you try and open a pull request to fix the documentation? This document should help and we're happy to walk you through anything else.

Thanks!

[byjrack]

And not sure what technology you are using for a Cache, but based on that linked issue JFrog confirmed that Cask was out of scope for their support because they can't act as a simple forward proxy and updated their docs to note that.

I believe the https://docs.brew.sh/Manpage#environment doc is still accurate though on what that envvar does to a download URL. Users may not understand though that Casks are sourced from any internet destination vs the Bottles by default sourcing from Github Packages. So you can swap out the source domain for Packages easily while the Cask payload requires a forward proxy that handles arbitrary destinations.

I still have not found a great combination of envvars that allows brew to leverage a cache for formulae and bottles while falling back to Internet access channels (forward proxy) for the payload of a cask. Didn't get much traction on a discussion of alternatives to the existing flow either so just been having users manually unset to install/update casks which is far from ideal.

[MikeMcQuaid]

so just been having users manually unset to install/update casks which is far from ideal.

I'd suggest this behaviour could be adjusted so it just doesn't take affect for Casks. I'd review a PR for that.

[byjrack]

Yeah that was the plan and just wanted to get some community feedback via the Discussion on what type of mechanism would be the best path. There is a bit of history in those existing variables and flows and wanted to try and reflect that. Ruby is also not my wheelhouse so that was a personal headwind that I know I could get by for a simple NO_CASKS flag, but the FORMULAE env vars it gets a bit more involved.

JFrog hadn't considered Casks in their initial implementation of the caching logic, but once I showed them how the flows were different than the Formulae/Bottle type interactions they knew they couldn't offer that while maintaining the controls in Artifactory.

I might take a swing at a simple PR on a to see how it goes. Couple internal hurdles, but nothing crazy.