Investigate and identify ways to reduce .NET build drop size

[mmitche]

The .NET build drops are huge and creeping upwards in size. We need to identify where the size is coming from and develop a plan to reduce that size, or we will start to see problems with downloading the drop and creating the layouts.

We should:

• Identify the total size of the download
• Identify the total size on disk, which may be different due to extra copies of files
• Identify artifacts that are duplicated on the whole in the download and on disk
• Identify artifacts that are duplicated within other containers (e.g. foo.dll in more than one package)
• Identify strategies to reduce the total size.

Not only is the large size a problem when attempting to complete the staging build due to disk space issues, but the download/artifact upload size has implicates for official and staging build performance.

[mmitche]

Added a switch to investigate the drop size here: #2785.

Initial findings from 186194:

• Download size - ~62GB
• Size on disk (after adding some copies of things) - ~83GB
• Duplicate size of top level assets (e.g. extra copies of things, or two packages with the same content) - 21GB
• After unpacking all zips, nupkgs, and targz files, Total size of duplicated files within the product - 179GB

There are some obvious low hanging fruits:

• There are many places where we generate symbol nupkgs that are identical to the original nupkg. Usually VS.REdist or source build intermediates that are just contianers of containers
• We could move to .snupkg and remove binary duplicates for symbol purposes altogether.
• We generate the package release layout in the build. This duplicates a bunch of artifacts and adds 10s of GBs to the overall size.
• There is a lot of duplication of llvm binaries Wasm packages, including transport packages.

[tkapin]

This seems to be a larger effort, probably worth bumping to an epic. Is this something we need to address in the .NET8 timeframe?

[mmitche]

No, though some improvement wouldn't hurt. I'll probably put in a fix for empty symbol packages in .NET 8 arcade.

[mmitche]

Actually, I think there are some epics here, and then some repo issues. I don't think all the work should go under a build drop epic:

• just issues:
  • Investigation of potential improvements to duplication in Wasm packages (product issue)
  • Avoid generation of symbol nupkgs that are identical to original nupkg (arcade? It may be arcade, but a lot of the generated packages are VS.Redist packages so that may be repo specific infra. Needs more investigations).
• Epics:
  • Snupkg support in arcade (arcade epic)
  • Remove the release package layout (staging/release pipeline epic).

Reason being is that the epics should be done even aside from build drop size concerns. And the issues aren't big enough to be their own epic.

[mmitche]

dotnet/arcade#1959 is the snupkg issue