How to change the default registry cache location

[rashton-snl]

EDIT:
As was pointed out in the comments below, I misunderstood what was happening. A more accurate question would be: How do I change the location of the default registry cache when using manifest mode?

Describe the bug

I am trying to use vcpkg in manifest mode (with CMake) in a Gitlab CI pipeline. In that situation, vcpkg fails, because it looks for packages in %LOCALAPPDATA%, which resolves to C:\Windows\system32\config\systemprofile\AppData\Local, which has restricted permissions. I found two relevant issues:

• gitlab CI runs as a service under user SYSTEM and binary caches aren't reused #35638
• Regression - vcpkg caches binaries in C:\Users\%username%\AppData\Local\vcpkg\ #16100

In both issues, it was recommended to set VCPKG_DEFAULT_BINARY_CACHE and VCPKG_BINARY_SOURCES, but that did not solve my problem.

As I was investigating this issue, I noticed very different behavior in how classic mode and manifest mode build packages. In classic mode, vcpkg behaves as expected. I can point it to a binary cache directory with VCPKG_DEFAULT_BINARY_CACHE or VCPKG_BINARY_SOURCES, and it uses that directory. When it builds a package, it never touches %LOCALAPPDATA%. When I try the same thing in manifest mode, it always puts ports in %LOCALAPPDATA%, regardless of how I set VCPKG_DEFAULT_BINARY_CACHE and VCPKG_BINARY_SOURCES. Like I said, that won't work for me due to directory permissions.

In my case, I suspect two possibilities:

1. I have completely misunderstood how vcpkg installs/builds ports, or
2. manifest mode ignores VCPKG_DEFAULT_BINARY_CACHE and VCPKG_BINARY_SOURCES when building ports.

Environment

• OS: Windows 11 Enterprise version 22H2
• Compiler: N/A

Expected behavior

I would expect VCPKG_DEFAULT_BINARY_CACHE and VCPKG_BINARY_SOURCES to affect manifest mode in the same way as classic mode. There should be some way for me to tell vcpkg not to do anything in %LOCALAPPDATA%.

Additional context

I have tried this on multiple computers under different user accounts with the same outcome. I have completely wiped everything related to vcpkg and reinstalled it. In the steps to reproduce below, I call vcpkg directly, but I see the same behavior when using vcpkg with CMake.

To Reproduce

Note that I set VCPKG_DEFAULT_BINARY_CACHE to $PWD\.cache\vcpkg\archives in all of the following cases, though I don't know that it matters, given the way that I set the binary sources.

--- Classic mode ---

Classic mode works as expected. At no point does it go looking for things in %LOCALAPPDATA% or %APPDATA%.

vcpkg install --binarysource="clear;files,$PWD\.cache\vcpkg\archives,readwrite" zlib

--- Manifest mode with existing cache ---

Now let's say I leave in place the cache created by the classic mode install, and I try to install in manifest mode:

vcpkg remove zlib
vcpkg new --application
vcpkg add port zlib
vcpkg install --binarysource="clear;files,$PWD\.cache\vcpkg\archives,readwrite"

It goes looking for packages in %LOCALAPPDATA%:

The following packages will be built and installed:
  * vcpkg-cmake:x64-windows@2023-05-04 -- C:\Users\%USERNAME%\AppData\Local\vcpkg\registries\git-trees\88a7058fc7fa73a9c4c99cfcae9d79e2abf87a5a
    zlib:[email protected] -- C:\Users\%USERNAME%\AppData\Local\vcpkg\registries\git-trees\3f05e04b9aededb96786a911a16193cdb711f0c9

It installs vcpkg-cmake from %LOCALAPPDATA%:

Installing 1/2 vcpkg-cmake:x64-windows@2023-05-04...
Building vcpkg-cmake:x64-windows@2023-05-04...
C:\Users\%USERNAME%\AppData\Local\vcpkg\registries\git-trees\88a7058fc7fa73a9c4c99cfcae9d79e2abf87a5a: info: installing overlay port from here

and it finds zlib in the specified cache:

Restored 1 package(s) from C:\Users\%USERNAME%\test-cache\.cache\vcpkg\archives

If I repeat this process:

Remove-Item -Path .\vcpkg_installed\ -Recurse
vcpkg install --binarysource="clear;files,$PWD\.cache\vcpkg\archives,readwrite"

then it finds both vcpkg-cmake and zlib in the cache:

Restored 2 package(s) from C:\Users\%USERNAME%\test-cache\.cache\vcpkg\archives

I don't know why it goes looking for packages in %LOCALAPPDATA%, but at least it does seem to end up pulling from the previously created cache.

--- Manifest mode without existing cache ---

Now let's get rid of that cache and try manifest mode again:

Remove-Item -Path .\vcpkg_installed\, .\.cache\ -Recurse
vcpkg install --binarysource="clear;files,$PWD\.cache\vcpkg\archives,readwrite" --debug

The problem happens here:

The following packages will be built and installed:
  * vcpkg-cmake:x64-windows@2023-05-04 -- C:\Users\%USERNAME%\AppData\Local\vcpkg\registries\git-trees\88a7058fc7fa73a9c4c99cfcae9d79e2abf87a5a
    zlib:[email protected] -- C:\Users\%USERNAME%\AppData\Local\vcpkg\registries\git-trees\3f05e04b9aededb96786a911a16193cdb711f0c9
[...]
Building vcpkg-cmake:x64-windows@2023-05-04...
C:\Users\%USERNAME%\AppData\Local\vcpkg\registries\git-trees\88a7058fc7fa73a9c4c99cfcae9d79e2abf87a5a: info: installing overlay port from here
[...]
Building zlib:[email protected]...
C:\Users\%USERNAME%\AppData\Local\vcpkg\registries\git-trees\3f05e04b9aededb96786a911a16193cdb711f0c9: info: installing overlay port from here

If vcpkg is running in a Gitlab CI pipeline, then it fails with the following error:

CMake Error at scripts/ports.cmake:138 (message):
  Cannot find port: vcpkg-cmake
    Directory does not exist: C:/Windows/System32/config/systemprofile/AppData/Local/vcpkg/registries/git-trees/88a7058fc7fa73a9c4c99cfcae9d79e2abf87a5a

The directory does exist, but vcpkg cannot build the port due to permission restrictions.

How can I tell manifest mode not to do anything in %LOCALAPPDATA%?

[Osyotr]

Directory does not exist: C:/Windows/System32/config/systemprofile/AppData/Local/vcpkg/registries/git-trees/88a7058fc7fa73a9c4c99cfcae9d79e2abf87a5a

This is not local binary cache. This is registry cache.
The probable reason it works for you in classic mode but not in manifest mode is because former uses built-in registry and latter uses git registry (set in your manifest).

[rashton-snl]

Thank you for your help, @Osyotr. As I suspected, it was a misunderstanding on my part. Is there a way to change the default registry cache location?

[Thomas1664]

The directory does exist, but vcpkg cannot build the port due to permission restrictions.

Which directory does exist?

C:\Users\%USERNAME%\AppData\Local\vcpkg\registries\git-trees\88a7058fc7fa73a9c4c99cfcae9d79e2abf87a5a or C:/Windows/System32/config/systemprofile/AppData/Local/vcpkg/registries/git-trees/88a7058fc7fa73a9c4c99cfcae9d79e2abf87a5a?

The later one seems very unusual to me and I highly doubt that vcpkg was able to put something in a subdirectory of System32.

[rashton-snl]

Surprisingly, the latter path does exist. The Gitlab CI pipeline runs under the system profile, and so %LOCALAPPDATA% expands to C:\Windows\System32\config\systemprofile\AppData\Local. Somehow, vcpkg is able to create that registry cache, but then it cannot access that cache when building the port.

The first path you list comes from my local tests outside of the CI pipeline (replace %USERNAME% with actual username) where I was trying to figure out a way to point vcpkg away from %LOCALAPPDATA%.

[dg0yt]

The Gitlab CI pipeline runs under the system profile

I consider this broken, but you will find more tickets.

[rashton-snl]

The Gitlab CI pipeline runs under the system profile

I consider this broken, but you will find more tickets.

Yeah, it's really not great how Gitlab runs pipelines under the system profile. I found several similar issues, but they all point to binary caching issues, which is what led me down that path. As Osyotr pointed out, though, this seems to be a registry cache issue. Ideally, vcpkg would provide some way for me to change the registry cache location, but I can't find any documentation to support that route.

[rashton-snl]

As a workaround, I temporarily point $Env:LOCALAPPDATA to a location with read/write permissions in my CI script. It's not pretty, but it does work.

Inspired by #12285

[ypogribnyi]

Registries cache location can be changed through X_VCPKG_REGISTRIES_CACHE environment variable, just set it to the absolute path of your directory. This var is not documented, I'm not sure why.

You might also want to set VCPKG_DOWNLOADS, X_VCPKG_ASSET_SOURCES and VCPKG_DEFAULT_BINARY_CACHE to make sure vcpkg doesn't use the %LocalAppData% directory.

[rashton-snl]

Registries cache location can be changed through X_VCPKG_REGISTRIES_CACHE environment variable, just set it to the absolute path of your directory. This var is not documented, I'm not sure why.

Thank you, @ypogribnyi, that worked!