Clear intermediate files after successful packaging

[ras0219-msft]

After building every (currently 90, wow!) port for static and dynamic, the combined size of installed, packages, and buildtrees is over 30GB. 23GB of this is because of the buildtrees, which are mostly intermediate files (but also sources).

We should keep sources around for optimal debugging, but we can easily shave off >70% by removing the intermediate directories (typically <triplet>-rel and <triplet>-dbg).

[adam4813]

Seems like running this type of command at the end after vcpkg_copy_pdbs.cmake or similiar finalizing would do the trick.

[Farwaykorse]

For the general user none of the buildtree files would be needed. And for port debugging, you'd want to have access to the files generated by Ninja (in the intermediate directories) too.

Suggestion

1. By default, after every successful build remove everything, except for the .log-files, from .\buildtrees\PORT\
2. Adding an optional flag to the install command for port development. Resulting in the current default behaviour and keeping all files used deploying the library.

vcpkg install --keep-all <pkg>

3. A second flag. To keep the extraction folder after successful installation. This would be usable for frequent rebuilds and while installing multiple configurations.

vcpkg install --keep-source <pkg>

This leaves the packages and download caches. All the rest can be regenerated when needed without external requirements.

Removing the triplet directories could be done in vcpkg_copy_pdbs.cmake.
This would be enough for most projects since header only libraries would not create these directories.
But that is assuming there won't be static only versions of libraries.
And that port implementers always execute this function even when building a static library.

Ideally such operations should not depend on the port-file implementation. The focus there is to get the library working. Clean-up should happen by default at closing.

[genbtc]

Agreed, we need something like this.

[cmpute]

Agreed as well, feel pain when moving whole vcpkg directory ... so many files!

[lua9520]

Agreed, vcpkg is too large after install some package.

[SergioRAgostinho]

What are the consequences of manually erasing the contents of buildtrees? I guess debugging operations no longer have sources to rely on, but other than that is there anything I should be aware off?

[MathiasMagnus]

Installing LLVM in just one config (x64-windows) has a 60GB+ buildtrees folder. I had to clear so many things to make the installer finish.

[Farwaykorse]

@SergioRAgostinho in my experience there are no issues with removing (most of) the content from the downloads and buildtrees folders.
Just don't delete the content of downloads/tools. Although I expect these will be restored at first use.

When updating already installed ports and adding new ports nothing changes.
Installing an additional triplet for a port or reinstalling will result in re-downloading and extracting of the source.
Deleting the archives from vcpkg/downloads, results in re-downloading if retrieved as an archive.
When the content of vcpkg/buildtrees was deleted, then the source will be extracted or downloaded from its repository.

It might be worth it for anyone to at least clean these folders up.
The downloads folder is collecting archives of old source versions.
Similar, buildtrees/port-name/src retains the source from all previous installs.

[MathiasMagnus]

Just as a sidenote, due to having so many duplicate copies during the build process, while adding features to the LLVM package, adding just AMDGPU and NVPTX (2 extra targets I need) I cannot finish the install, because 100 GBs is not enough. This is crazy. I'll have to relocate my Vcpkg install to my HDD, because a 256 GB SSD is not enough for Windows and dev tools only (no media, no games).

[MathiasMagnus]

I thought I'd share a workaround for the excessive storage usage. In case you have 2 drives in your system (I have an SSD where my work, including source code resides and an HDD where media resides) you could create symlinks for the intermediate directories. In my case it slowed down builds (HDD vs. SSD) but at least I could build LLVM. For example in PowerShell

Remove-Item -Recurse C:\Users\mnagy\Source\Repos\vcpkg\downloads
New-Item -Type Directory D:\vcpkg-tmp\downloads
New-Item -Type SymbolicLink -Path C:\Users\mnagy\Source\Repos\vcpkg -Name downloads -Value D:\vcpkg-tmp\downloads\

And you should do the same for the buildtrees and packages directories. This way resulting binaries for further builds are fast and only those reside on your primary drive.

[MikeGitb]

Any update on this?

[BillyONeal]

We currently have a --clean-after-build switch that achieves this effect, but it also causes PDBs and other intermediates that might be important for some ports to be clobbered.

[PhoebeHui]

I think '--clean-after-build' has already implemented the origianl plan.