Exclude entire packages from single-file

[iq2luc]

The problem

Actual situation

Unreasonably "fat" executables (due to code redundancy) when deploying multiple single-file executables (e.g. lots of small console based tools) that all depend on a set of shared libraries (part of them coming from nuget packages).

Expected situation

An easy way to exclude entire nuget packages (and their implicit dependencies) from single-file deployment.

Solution proposals

Solution 1

Allow ExcludeFromSingleFile to work on PackageReference similar to how it works on ProjectReference. Also, it should affect all implicit dependencies.

or

Solution 2

An option to generate a single-file (apphost + main assembly + required jsons) without embedding any other referenced assemblies. Then the user may opt-in for embedding specific assemblies via IncludeInSingleFile for a PackageReference and / or ProjectReference. Basically this would be the logical inversion of Solution 1.

Additional context

Just an example regarding my particular use case.

I have several (8 for the moment) small(ish) console based tools (~ 150 KB - 1 MB / program), all depending on a set of shared libraries (~ 10 MB in total). I'm doing single-file framework-independent deployments because I cannot enforce the client to do a system wide install of the .NET framework. The actual deployed package is just an archive with all the tools together, the client just unpacks it and uses the tools. The sad thing is that the useful code for each program is just insignificant noise compared to the actual file size.

Some stats on various publishing options:

• --self-contained -p:PublishSingleFile=true
  Each tool is ~ 70 MB
  Everything is ~ 550 MB

• --self-contained -p:PublishSingleFile=true -p:PublishTrimmed=true
  Each tool is ~ 25 MB
  Everything is ~ 200 MB

• --self-contained -p:PublishSingleFile=true -p:PublishTrimmed=true -p:EnableCompressionInSingleFile=true
  Each tool is ~ 20 MB
  Everything is ~ 150 MB

• --self-contained -p:PublishSingleFile=false
  Each tool is ~ 150 KB - 1 MB (as desired)
  Everything is ~ 80 MB
  A lot of files to deploy, a chaotic mix of native shared libraries, assemblies, executables and jsons

In the end I'm using the last variant combined with some hacks. I build each project separately with PublishSingleFile=true so the generated executable won't require json files alongside them, but also don't embedded all shared libraries (i.e. don't use nuget packages, only explicit assembly references, set ExcludeFromSingleFile accordingly etc.). On top of that, adding insult to the injury, I had to do a dirty little hack by patching libhostfxr.so to workaround a bug that - while simple by itself - it was critical for a console program (see dotnet/runtime#64606, many thanks to @vitek-karas for already fixing it).

Is it possible (or could it be made possible in the future) for the self-contained deployments to have all required framework specific files (native and managed libraries) in a sub-directory (e.g. ./CoreLibs), all business specific files (native and managed libraries) in another sub-directory (e.g. ./UserLibs) and keep the base directory (e.g. ./) "clean" with only executables in it?

[baronfel]

The runtime assets for a package/project reference are copied to the build/publish directories, so if you have no need for those since you're supplying them through some other mechanism, I think you could get there with ExcludeAssets="runtime" on the specific package references you desire. This is what we suggest that MSBuild Task authors do, since when the tasks are executed by MSBuild the MSBuild Host is responsible for supplying the runtime implementation of the abstractions/utilities libraries, for example.

[vitek-karas]

Part of this sounds a lot like: dotnet/runtime#64430. Which is about the ability to share a private runtime install among several apps. The idea there is that the application's executable would have a hardcoded relative path to the location of the runtime - the runtime in that location would be a normal private install (xcopy install).

Another part is about "Shared libraries" which is something we're thinking about, but hasn't really come up with a good solution.

The immediately actionable part is about adding the ability to specify ExcludeFromSingleFile metadata on a PackageReference and have it applied to all assets from that package (and all its dependencies?, this might be tricky). I assume this already works on ProjectReferences, if not it's a bigger problem. And there might be some complexity in handling transitive dependencies. Especially if there are conflicting rules (package A is "exclude", package B is "include" both depend on package C... so what should C be?)

[iq2luc]

Indeed, the second part of the post (i.e. "additional context") is basically a similar ideea as dotnet/runtime#64430 and more so as dotnet/runtime#53834. I just described my use case and the issues I encountered. Sorry for not properly searching for a similar issue first.

I assume this already works on ProjectReferences [...]

Unfortunately no, it doesn't seem to work. If we have Exe project dependent on LibFoo project which depends in turn on LibBar project, specifying ExcludeFromSingleFile for LibFoo in the Exe project doesn't exclude LibBar -- it is still integrated into the final single-file blob (unless we explicitly add to Exe an ExcludeFromSingleFile option for LibBar).

That's why I mentioned ExcludeFromSingleFile should ideally be transitive and affect implicit dependencies. Although, I suspect it is indeed hard / tricky to do it right.

On the other hand, it doesn't work at all on PackageReference. While it would be great to "just work", I guess it adds a lot more complexity compared to ProjectReferences. Working similar to ProjectReferences (exclude all assemblies from the package without considering the dependencies) would be a small but significant step forward though.

[LaraSQP]

I second this request.

ExcludeFromSingleFile should work regardless.

[richlander]

I wrote a brain-dump spec on this topic some time back: https://github.com/dotnet/designs/blob/shared-libraries/accepted/2021/shared-libraries.md. Some of the ideas in this doc are also realted: https://github.com/dotnet/designs/blob/main/accepted/2022/version-selection.md.

We need a way to say "look here for other stuff" in a way that makes sense. The ExcludeFromSingleFile idea is good, but it seems to solving only one half of the problem. Also, we already have support for frameworks in our toolchain. I'm hoping we can rely on it as part of the solution for this problem.

[liesauer]

@iq2luc you can try NetBeauty2 which supports shared runtime by multiple apps, i think that is what you need.

[iq2luc]

@liesauer Thanks for your suggestion, I already started working on something similar, plus bundling assemblies together (even in the executable itself if needed) as compressed (brotli or lz4) streams. So far I have the executables bundled with the assemblies used only by them, the shared assemblies bundled together by scope in a sub directory, and the actual framework files (assemblies and shared objects) inside another sub directory (some of the assemblies bundled too). Unfortunately, for the moment I have a lot of manual steps for an actual deployment, not to mention the "hack-iness" of it (including dirty patching here and there).
All in all, I think I need to take a closer look at the hosting components source code and come with a custom solution for for my use case.

Still, the scope for the current issue is just to make it possible to exclude package and project references from publishing (including dependencies). That one, together with embedding json files would get us closer to the long term goal of having a "beautiful .NET deployment", as your project implies. :-)

[nietras]

We have some very large nuget packages, that we don't want to embed into a single file application, so was sad to see there apparently is no way to actually do that using current approach. I hope this is revisited for .NET 8 time frame, but hope there is some kind of workaround? Perhaps via https://learn.microsoft.com/en-us/dotnet/core/deploying/single-file/overview?tabs=cli#post-processing-binaries-before-bundling one can make a target where particular files can be removed from FilesToBundle?

[nietras]

Using LLM I came up with a way to exclude certain nuget package dlls as part of build by:

  <Target Name="ExplicitRemoveFromFilesToBundle" BeforeTargets="GenerateSingleFileBundle" DependsOnTargets="PrepareForBundle">
    <ItemGroup>
      <FilesToRemoveFromBundle Include="@(FilesToBundle)" Condition="$([System.String]::new('%(Filename)').ToLower().Contains('.model.')) AND ('%(Extension)' == '.dll')" />
    </ItemGroup>
    <Message Text="FilesToRemoveFromBundle '@(FilesToRemoveFromBundle)'" Importance="high" />
    <ItemGroup>
      <FilesToBundle Remove="@(FilesToRemoveFromBundle)" />
    </ItemGroup>
  </Target>

  <Target Name="CopyFilesToRemoveFromBundle" AfterTargets="Publish">
    <Copy SourceFiles="@(FilesToRemoveFromBundle)" DestinationFolder="$(PublishDir)" />
    <Message Text="Copied files to remove from bundle to '$(PublishDir)'" Importance="high" />
  </Target>

"It should remove the files that contain “.model.” and end with “.dll” from the single file bundle and copy them to the publish directory."

[slxdy]

Bump. The fact that we still have to use work-arounds for this is ridiculous.