Publishing to Single-file in .Net Core

[swaroop-sridhar]

The is a design proposal for supporting single file distribution in .Net Core.

To run apps published as a single-file, CoreCLR#20287 proposed extracting out all the embedded dependencies to files at startup.

However, based on customer feedback through public and private fourms, the current design uses a hybrid of in-memory execution and extraction to files.

Further stages of improvements are outlined in staging.md.

[ericsampson]

For self-contained, what happens the second (and subsequent) runs of the .exe? I'd hope that it would be smart enough to only extract the files the first time.

[swaroop-sridhar]

@ericsampson the proposal to reuse extracted files on subsequent runs is discussed here: https://github.com/dotnet/designs/pull/52/files#diff-5e93215a94e626495c10448a40fa4ccbR40

[ericsampson]

Thanks @swaroop-sridhar I did end up finding that, it just wasn't mentioned in the design.md user User Experience which makes it seem like the extraction was always going to happen each time: Cheers!

Run: HelloWorld.exe

The bundled app and configuration files are processed directly from the bundle.
Remaining 216 files will be extracted to disk at startup.

[swaroop-sridhar]

Thanks for pointing that out @ericsampson. I added a note about it here

[ayende]

One thing that I haven't seen addressed is the issue of a platform dependent native libraries.
I assume that this bundled apps are per platform anyway, but how do I select which native deps I'll bundle with each?

[ayende]

What about embedded symbols in the file? I assume that would still work as usual, but the text make is seems like that would be separate?

[swaroop-sridhar]

Yes, I was only talking about the fact that PDB file will be separate and not embedded within the single-file app. I'll change the wording.

[ayende]

That is actually a concern for me. I want to be able to have a single package that include the PDB. We currently do that with embedded pdbs, and I assume this will continue to work?

[jeffschwMSFT]

Thanks for the feedback, we could make this configurable. I think by default (to save size on disk) this would be off.

[swaroop-sridhar]

@ayende you're asking about assemblies generated with debug info embedded (ex: csc -debug:embedded) right? Those are fine, I'll add it as a scenario to test.
If a separate PDB file is generated, we won't add it to the single-file. But as @jeffschwMSFT says, we can make it configurable if necessary.

[ericsampson]

+1 on configurably embedded PDB.

[ayende]

Yes, I am. If those work, I'm happy with that.

[ayende]

We'll also require a flag like IsBundled, to know how to behave, no?

[swaroop-sridhar]

My thought was that LoadWithEmbeddedDependencies will only be used with bundled assemblies.
I agree that more thought is required about the necessity/details of the LoadWithEmbeddedDependencies API, while implementing support for single-file plugins.

[jeffschwMSFT]

@ayende this is a good suggestion. Can you describe a potential scenario?

[ayende]

Consider the case of resources that are bundled into the app. In our case, we have a web app that has a lot of files (js, png, css, etc).
We actually have three separate options here right now. During dev, we might be working on the raw files, the compiled files or during release, loading them from a zip file.

Being able to tell how we are running would be helpful because if IsBundled == false, I need to probe a whole different location to see where the file I need to serve it located.

[jeffschwMSFT]

Good feedback, and thanks for the scenario details.

[swaroop-sridhar]

OK, thanks for the details @ayende. For this situation, I think it would be better served by providing an abstraction that handles bundles and files seamlessly, as proposed here.

But I think it is also reasonable to expose an API like the following, to support custom scenarios.

namespace System.Runtime.Loader
{
    public partial class Bundle
    {
        public static bool IsBundle(Assembly assembly);
    }
}

[ayende]

This implies that each launch will extract the files from scratch? This can be a pretty big cost, no?
Can we do this only once?
What about if we'll use the file hashes and use that? This helps both concurrent instances and 2nd run.

[swaroop-sridhar]

Reuse of extracted files is discussed in the next section; this section is about the option of extracting to temporary locations on every use.

[jeffschwMSFT]

We have discussed having reuse as an option. Both 'extract everytime' and 'reuse' have different pros and cons. We plan on leaving it up to the developer to choose.

[ayende]

I don't understand this statement

[swaroop-sridhar]

Click-and-run model: App runs with necessary extraction on startup and cleanup on exit. There are no side-effects after the app is run.
Install model: App installs the extracted files to user-visible non-temporary storage, for reuse across multiple runs.

The point here is that the temporary-extraction solution suits situations that don't prefer apps leaving behind state on disk.

[ayende]

The issue is that Windows has to load the native library from a file, right?
In the click and run scenario, can you write the files with FILE_FLAG_DELETE_ON_CLOSE, call LoadLibrary on the file and then close it?
That would mean that the OS will take care of cleanup automatically.

[swaroop-sridhar]

Thanks @ayende. We can consider that for implementation on Windows.
However, we will still need a clean up step on exit, because there may be files other files (ex: data files) that are bundled and extracted out to disk.

[ayende]

Can probably do that as well. Extract them, open with delete on close and "leak" the handle.
The OS will handle that for you.
On Linux, I think we can load a memfd_create and then dlopen , so we don't even need to touch the disk

[KathleenDollard]

The sentence construction is confusing: "don't prefer...rather than". I do not know what you are saying.

Can you restate this intent in different words.

[swaroop-sridhar]

Sorry I'll fix the typo.

[ayende]

I would love this option, because then the only files extracted would be the apps native dependencies, right?

[swaroop-sridhar]

Yes

[KathleenDollard]

This is going to be a great state. The approach of this proposal allows a step-wise approach so we don't have to have everything ready before we provide anything.

[ayende]

I would love that option. It would solve quite a lot of issues for us in deployment mode.

[swaroop-sridhar]

how do I select which native deps I'll bundle with each?

Yes @ayende Single-file apps are platform-specific. So, they must be generated from a platform specific publish (wrt a particular RID), whether framework-dependent or self-contained.

Embedding the correct native dependencies into a single-file is no different from copying the native dependencies to the RID-specific publish directory. The single-exe part of the build only embeds the contents of the RID-specific publish directory into the host.

Are you asking about adding additional native binaries that are not in the publish directory?

[MichalStrehovsky]

We might need to define what APIs like Assembly.Location and Assembly.CodeBase return for bundled assemblies. This is a problem that has been plaguing CoreRT/ProjectN for a long time (see e.g. dotnet/corefx#35148 (comment), dotnet/corert#5647, dotnet/corert#6679 (comment), dotnet/corert#6947, dotnet/corert#6946).

I haven't been able to find a solution that would work for everyone and the best I have is for us to offer a new AppContext switch to select one of these behaviors (one of them should be the default):

• Location returns an empty string. This is a documented behavior of Location if the assembly got loaded from byte array. Theoretically, user code should be able to handle that. In reality, this is a very good way to break everyone using this API.
• Location returns the name of the assembly, without a file path. This should work for people who want to roundtrip the location through e.g. Assembly.Load.
• Location returns the name of the assembly, qualified by AppContext.BaseDirectory. This makes existing code that does Path.GetFilePath(Assembly.GetEntryAssembly().Location) to get to the app's directory. This pattern is extremely common.

[ViktorHofer]

I don't have much context here but naively speaking, couldn't option 3 (AppContext.BaseDirectory + assembly name) also allow round-tripping with Assembly.Load? Means if the full path passed into Assembly.Load is the current executing assembly it won't attempt to load that assembly and just returns.

[jkotas]

This is a problem that has been plaguing CoreRT/ProjectN for a long time

I think the large part of the problem has been that CoreRT/ProjectN were niches and not many folks cared to make their projects compatible with the "single file" model used by them. The good news is that making "single file" model more first class citizen will make more folks to care about it.

We are going to need new APIs to allow apps and libraries to do what they need to do in the (bundled) single file mode. It is unlikely that we would be able to make things "just work" well by quirking existing APIs. I would worry about making sure that we have all the right API first, and only worry about quirking existing APIs last based on the feedback.

[nguerrera]

Is this at odds with the other goal of "Able to generate cross-platform bundles (ex: publish a single-file for Linux target from Windows)"? My understanding is that at present, we do not support "cross-crossgen", correct?

[swaroop-sridhar]

Thanks for pointing that out @nguerrera. Yes, we can generate cross-platform bundles without any new cross-compilation optimizations, until cross-crossgen is available.

[KathleenDollard]

@swaroop-sridhar is there a "not" missing from the second sentence? Is it that we cannot until we have cross-crossgen?

[swaroop-sridhar]

Yes that's a typo, sorry.

[nguerrera]

There are IDE considerations for adding new item types. This may not give the best experience for maintaining such files in the IDE.

We should consider if this can be metadata on Content. Today there is CopyToOutputDirectory, CopyToPublishDirectory. There can be IncludeInSingleFilePublish or something.

Today, CopyToOutputDirectory implies CopyToPublishDirectory by default and you can override.

[swaroop-sridhar]

That's a good suggestion @nguerrera, I'll change SingleFile Include/Exclude to be meta-data.

[swaroop-sridhar]

@nguerrera I've updated the doc now to use IncludeInSingleFilePublish metadata.

[nguerrera]

Should PublishSingleFile change the default output directory of the publish? Incremental publish today already has some problems. I suspect publishing alternating between single-file and not will compound them if we're not careful. We might also take the opportunity to spec out the incrementality better, fix bugs, and handle this case carefully.

[nguerrera]

Or we could not advertise using a global swtich on publish and instead recommend setting PublishSingleFile in project file based on configuration. In general, switching global properties between msbuild invocations is tricky business with respect to incrementality.

[swaroop-sridhar]

The output directory issue is should be handled similarly for all post-processing tools (currently ILLinker, crossgen, and single-file). The publish process may temporarily use other directories, but ultimately, the publish directory should contain the single-file, I think.

I agree with the idea of marking the PublishSingleFile on the project file rather than the command line, in order to reduce confusion wrt incremental build.

[nguerrera]

We may want to note the option to add dotnet publish --single-file as sugar for dotnet publish /p:PublishSingleFile=trure . Whether this is a good idea depends on how we answer my other questions above.

[ericsampson]

I know it seems really minor, but IMHO having CLI sugar for this would be really nice in terms of lightweight usage, demos, discoverability, etc.

[KathleenDollard]

Is there just the single switch?

Agree that we want some CLI love on this. Want to design it with the full richness we'll need later.

[MichalStrehovsky]

We are going to need new APIs to allow apps and libraries to do what they need to do in the (bundled) single file mode. It is unlikely that we would be able to make things "just work" well by quirking existing APIs.

Not having tools available to help existing code bridge the gap until it can move to the new API is always a source of sadness for the customer, and for the person trying to help the customer. I'm not as enthusiastic about libraries adapting to this quickly. Single file deployments will be niche, unless we make them the default.

I had conversations like this many times, and some of them were literally about Assembly.Location:

- Just don't call this API, call this other new API instead.
- The call is in a NuGet package.
- Oh.

[jeffschwMSFT]

@MichalStrehovsky thanks for bringing this perspective. The good thing is that in these cases these files could be spilled to disk. In general we strive to enable the spill to disk with high fidelity to the original apps intention.

[ericsampson]

Is someone from Azure App Service looped in this design discussion, because it could play in very nicely with Run From Package (which is also used by Azure Functions, so coldstart perf improvements if the 'extract once' functionality is leveraged automatically? Or eventually the Stage 4/5 'extract and run in-mem' could be huge?) @odhanson @jeffhollan

[swaroop-sridhar]

Thanks @ericsampson. That's a great idea.
@KetanChawda-MSFT can you please tag the team that handles "Run from package" app service, to take a look at this design? Thanks.

[MichalStrehovsky]

The good thing is that in these cases these files could be spilled to disk

They'll be spilled to the temporary location so the code that assumes it can read e.g. a config file from Path.GetFilePath(Assembly.GetEntryAssembly().Location) is still going to get some path in %TEMP%. We might as well return the same string we would return for the embedded case so that the implementation detail of extracting to a temporary location won't be visible (especially if we're extracting due to a current implementation limitation and not because the user asked for that).

It's not an academic scenario - we see this pretty often in ProjectN. Here's one in xunit.

[jeffschwMSFT]

@MichalStrehovsky why would we hide the paths? In general being single-exe is a chosen path and spilling to disk is also a choice. I would opt for defaults that enable the most things to 'just work'. I would presume that if we spill to disk and not hide the path that most uses of Location, etc. would just work. Are there cases that do not?

[MichalStrehovsky]

I would presume that if we spill to disk and not hide the path that most uses of Location, etc. would just work. Are there cases that do not?

The cases when the assembly is not interesting, only the directory where the assembly is. Path.GetDirectoryName(Assembly.GetEntryAssembly().Location) is a common way to get to the "app directory" to read config files or load plugins from. Doing that is roughly equivalent to calling AppContext.BaseDirectory right now. If we do nothing, it will either be an empty string (loaded from memory), or a path to the extracted files, depending on implementation details (that might change).

We either come up with a standardized way to approach this (e.g. single file deployments on Xamarin seem to prefer returning directory-unqualified assembly file name), or every single implementation is going to have their own way to do this, breaking user code in a different way (CoreCLR is not the first runtime to have single-file deployments in .NET). There should be just a single way to do this in .NET Core.

[poke]

One question I had that wasn’t answered: When extracting the files to a temporary or permanent location, writing to that location is an unsupported action, right? E.g. if I have an application with embedded configuration, and that configuration file then gets extracted somewhere, I should probably not edit that file but rather store the edited one somewhere else (e.g. next to the host application).

[poke]

I feel like configuration shouldn’t be included in this list by default. If I think about a common use case for this single-file system, then I am thinking of easy-to-distribute utility applications that likely come with a default configuration setup and can be further configured by a file that’s next to the application. If the configuration file gets embedded by default, then there needs to be additional logic to support an external configuration file (outside of the bundle).
But if the configuration file does not get embedded, this can be simply handled as the file being optional (this would be very easy to implement with M.E.Configuration).

[jkotas]

The problem is how you define "configuration". What is configuration for you may not be a configuration for me. As mentioned elsewhere in the doc, you will be able to customize what to include in the bundle.

[swaroop-sridhar]

By configuration, here I mean .net-core's configuration files which are recognized and handled by the .net-core host/runtime. All other files are data-files from the point of view of the runtime, and should be handled as such.

[jkotas]

E.g. ASP.NET comes with bunch of its own configuration .json files that are just data-files from the point of the runtime.

I do not think the distinction between data files and configuration files matters much. All publishes stuff will get bundled (correct?) by default, it does not really matter what it is.

[swaroop-sridhar]

Yes that's correct, Whatever is in the publish directory will get bundled, other files can be included too, as you mentioned above. Thanks.

[poke]

Don’t framework-dependent publishes only include the app’s .dll and to run it, you have to run dotnet HelloWorld.dll? Where does this .exe come from?

[jkotas]

That was changed for .NET Core 3.0. You get .exe for framework dependent publishes by default in .NET Core 3.0.

[poke]

Oh cool, didn’t know that! So the .exe will then pick up the “correct” runtime if one is installed automatically? Do you happen to have a link for me where this process is described?

[vitek-karas]

You can find most of this process described here

[poke]

What happens when plugins require a common dependency that would already be there with the application? Would they have to bundle it again?
For example, let’s assume application Foo.exe uses a library Foo.Common.dll. Now, plugins can be built that use the Foo.Common.dll. The application then has some mechanism to pick these up.

With the classic approach, or even .NET Framework, the application might have exactly those two outputs: Foo.exe and Foo.Common.dll. And when you create the plugin, you might end up with plugin.dll and Foo.Common.dll. Now since the common library would already be bundled with the application, distributions of the plugin would only have to contain the plugin.dll. You could paste that into the application folder, enable the plugin, and it would work.

With .NET Core and single-file publishing, this would now mean that both the application and the plugin would bundle Foo.Common.dll and all the runtime dlls into their final output. So you have those twice for this setup. Of course, you could opt in to make the library not bundle into a single file, but the plugin might also have some other dependencies, so it likely would want to bundle into a single file. So in the end, this would require a lot of configuration for the plugin author to produce an efficient output.

Do you have any thoughts on this situation?

[jkotas]

This problem of what dependencies to include with plugins is not specific to single .exe deployment. You are right that one needs to be aware what comes with the app vs. what needs be included with the plugin.

Take a look at plugin sample https://github.com/dotnet/samples/tree/master/core/extensions/AppWithPlugin that shows how it can be done.

[swaroop-sridhar]

I agree, the goal of this work is to just package the contents of the publish directory to a single file. It shouldn't change the contents of the publish directory. The way to resolve missing dependencies for a plugin depends on it's configuration and isolation settings.

For example, if we didn't package Foo.Common.dll with plugin.dll, and plugin.dll is loaded in it's own AssemblyLoadContext, then it could:

• Choose to not load Foo.Common.dll in its own ALC, in which case, the load-logic falls back to the default ALC (of Foo.exe) and loads Foo.Common.dll in that ALC.
• Choose to explicitly load Foo.Common.dll from the Foo.exe bundle, using GetFileStream API proposed here.

[poke]

This problem of what dependencies to include with plugins is not specific to single .exe deployment.

The difference for non single file deployments is that you can simply diff what files would be added by your plugin and only distribute those.

[…] the load-logic falls back to the default ALC (of Foo.exe) and loads Foo.Common.dll in that ALC.

So it would work if you deploy the app as a single file and choose not to do that with the plugin (distributing only the plugin.dll), and the plugin.dll could then still (automatically) access the common library and (more importantly) the framework dependencies from the Foo.exe?

The only problem I would see if the plugin has additional dependencies and wants to do a single-file deployment of only those dependencies that are specific to the plugin (and aren’t already included in the app). But I guess this would work with customization of the bundling behavior, so all the framework assemblies are not bundled.

On that note, I could actually imagine just bundling the framework assemblies for every single self-contained application, so that there’s only the .exe, the concrete dependencies, and then maybe a framework.dll. – Would that work somehow or would that already be problematic because the executing assembly needs to be a host file that is able to unpack the files?

[swaroop-sridhar]

@poke : A few points to clarify:

• The plugin and app should be developed with a proper expectation of each other's dependencies (native binaries and assemblies).
• If diffing published files is desirable, we still have the option to publish without the single-file option. The dependencies should not be different whether we publish single-file or not.
• Framework is typically never packaged with plugins.
• The runtime will provide call-backs extract file embedded within plugins. We may be able to refactor the host code such that the callbacks may reuse that code for handling single-file plugins.

[poke]

Would this be configurable? I could imagine that some applications might rather have this be %TEMP%\MyCompany\<random-code>.

[swaroop-sridhar]

The .net part just signifies that these are temporary files created by .net-core. This way, the entire .net folder can be easily cleaned up if necessary. We could, of course, have %temp%\.net\app\... etc. But we choose shorter file names -- so that we don;t hit the max-path limit.

[poke]

Yeah, I do understand the idea behind this but I could still see some situations where having a .net in the path would be unwanted. The fact that the application is a .NET application is an implementation detail after all.

[poke]

Typo: ”writes an end-marker”

[poke]

For self-contained apps, users are not expected to have a dotnet CLI, so there might be some other approach needed to clean it up. Would it be possible for the app to clean up itself, to implement some “Exit and uninstall” functionality into the application?

[swaroop-sridhar]

You mean not cleanup on every exit, but on a specific uninstall run, right? Yes, the app-developers can implement this, because the install-location of extracted files is predictable.

[poke]

Yeah, using a persistent location (so that the extracting only happens once) but at the same time offering an option within the application to remove the extracted files at that location.

[poke]

In extract.md, the temporary extraction was explained to clean up the temporary location on exit, so how does this work with this explanation where there the extracted files are reused?

And how exactly would users be able to identify no longer needed files? Especially when running an application through multiple versions, I would imagine that it would be quite difficult for users to identify which version they are running and which files they can delete (especially since this would still be a manual process).

[swaroop-sridhar]

Removal of persistent-extracted files will be manual in this version, as noted here. That is, user can look for the app in $HOME/.dotnet/Apps/<AppName> and delete the apps he doesn't want to use anymore.

Here, versioning is based on a hash-of the entire single-file (including assemblies, binary and data files, signatures etc). That is, if the app, or any of its dependencies changed when recompiled, we will extract to a new location. So, it is tricky to provide an accurate human-readable description generically for all apps.

However, we can provide some information to help this situation. For example, as part of the end-marker, we can write out the full name and assembly version of the app's main assembly (or version of all assemblies in the app).

[poke]

What about an option that combines these two approaches? I.e. extract the native libraries to the current folder on first launch (and whenever they get deleted).

[swaroop-sridhar]

This option can be achieved by using persistent-extraction, and setting the extraction-location as the current directory, using the configuration mentioned here.

[ayende]

WRT Location, what about /home/me/app.bundle;app.dll ? That would still maintain the get the base direction method, and it is also clear that this is something new.

[ayende]

About config files, etc.
I can tell you what we would do in this case. We currently have a settings.default.json file that we bundle with our distribution. At startup, we search first for settings.json in the base directory and copy the settins.default.json file.

We'll absolutely need that to be in the same directory as the bundled app for discovery to the user.

On the other hand, I can certainly see the need for throwing some data in such an bundled app somewhere that isn't clearly visible (~/.myapp) works great and there are already APIs for it.

[swaroop-sridhar]

@MichalStrehovsky I added the discussion about Assembly.Location here.
I think Assembly.Codebase can remain unchanged for bundled or normal assemblies.

[jkotas]

Also: dotnet-warp global tool

(mentioned in https://www.hanselman.com/blog/BrainstormingCreatingASmallSingleSelfcontainedExecutableOutOfANETCoreApplication.aspx)

[nsentinel]

It may make sense to look at the https://github.com/MiloszKrajewski/LibZ design with similar goals for the .NET Framework.

We have used it for a long time without any issues in many projects (thanks to @MiloszKrajewski)

[swaroop-sridhar]

Thanks for the reference @nsentinel.
The core of LibZ is to embed assemblies as resources, and instrumenting the app to use resolvers to load the assemblies from resources. LibZ also adds a container abstraction to reduce assembly size.
I'll add LibZ to the list of related work. Thanks.