How to get single file extraction path in .NET 5.0?

[tomrus88]

In .NET Core 3.1 it used to be AppContext.BaseDirectory, but that seems to be changed in .NET 5.0 to return a path where bundle executable is instead of extraction path.

[ghost]

Tagging subscribers to this area: @agocke
See info in area-owners.md if you want to be subscribed.

[agocke]

By default, in 5.0 the binaries are not extracted at all, so there is no extraction directory.

[tomrus88]

But most of single file apps have native dependencies that will be extracted if you use IncludeNativeLibrariesForSelfExtract option (and 99% of users are gonna use this option, because they want single file). So in nearly 100% of cases something will be extracted and will never cleaned up because there's no way to get path to extraction folder...

Blank WPF app is extracting 9 native dll's with total size of 16MB. That means with each update an additional 16 MB of disk space will be wasted. And this is just blank app... Some real world app can have much more native dependencies and waste much more disk space with each update...

[Fabi]

By default, in 5.0 the binaries are not extracted at all, so there is no extraction directory.

Native binaries are ALWAYS extracted.

[tomrus88]

Another example where extraction folder path is required: let's assume I want to include some data file (I need to have read/write access to this file, so resources is not an option) with my app and bundle in into single file, this file will be extracted along with all native dependencies to unknown folder. How do I specify path to this file in the code so I can work with data in that file?

[iSazonov]

Related comment #43014 (comment)

https://github.com/dotnet/designs/blob/main/accepted/2020/single-file/design.md in .NET 5 anything except managed code will not be bundled in the application and should be left next to the main .exe.

[tomrus88]

Leaving everything but managed code next to exe will defeat whole point of using single file.

[Fabi]

Related comment #43014 (comment)

https://github.com/dotnet/designs/blob/main/accepted/2020/single-file/design.md in .NET 5 anything except managed code will not be bundled in the application and should be left next to the main .exe.

It's only half related to that imo. Having native binaries not bundled into a bundle is really stupid design and there is NO reason to not implement loading native binaries from memory. It's possible, not hard to do and works fine. It's probably not even much effort to implement it in the current bundle process.

[MichalStrehovsky]

there is NO reason to not implement loading native binaries from memory

Do you have pointers on how to do that? Windows doesn't have APIs for this. .NET 5 no longer extracts on Linux because there it's possible.

Handwritten PE loaders, while appearing to sort-of-work don't play well with the rest of Windows so things end up being broken in subtle and hard to troubleshoot ways (from unimportant things like the module not being visible in tools like Process Explorer, through not being able to debug the module with native debuggers, etc.).

[agocke]

I want to include some data file (I need to have read/write access to this file, so resources is not an option) with my app and bundle in into single file

Our recommendation here is that you should write that file yourself to a temporary folder. This is much more robust and portable.

[Fabi]

there is NO reason to not implement loading native binaries from memory

Do you have pointers on how to do that? Windows doesn't have APIs for this. .NET 5 no longer extracts on Linux because there it's possible.

Handwritten PE loaders, while appearing to sort-of-work don't play well with the rest of Windows so things end up being broken in subtle and hard to troubleshoot ways (from unimportant things like the module not being visible in tools like Process Explorer, through not being able to debug the module with native debuggers, etc.).

I'm sure that mirroring LoadLibrary from in C# is no issue nowadays and works fine if implemented correctly. And that part would only be required for the actual native library that is shipped. All other native libraries that are required (like kernel32 etc.) for doing this can still be loaded with calls like LoadLibrary. I think there are already a bunch of libraries that even support doing that. Another way is to do it as runtime support with a c(++) implementation of course.

[Fabi]

I want to include some data file (I need to have read/write access to this file, so resources is not an option) with my app and bundle in into single file

Our recommendation here is that you should write that file yourself to a temporary folder. This is much more robust and portable.

This doesn't solve the issue of being able to cleanup data on app updates tho.

[agocke]

This doesn't solve the issue of being able to cleanup data on app updates tho.

Why not? It's not automatically cleaned up for you, but it's a file like any other. You can delete it if you want.

[tomrus88]

Why not? It's not automatically cleaned up for you, but it's a file like any other. You can delete it if you want.

And there we come to original problem of API that returns extraction path being unavailable...

[Fabi]

This doesn't solve the issue of being able to cleanup data on app updates tho.

Why not? It's not automatically cleaned up for you, but it's a file like any other. You can delete it if you want.

What tom said!!!111!

[agocke]

Sorry, I'm slow this morning, I think I finally understand the scenario.

You want to update by downloading the new app, then pass the extraction directory to the new app when you invoke it, so it can clean up the old directories.

Unfortunately, if you're using only IncludeNativeLibrariesForSelfExtract, there's no supported way to access the native libraries aside from P/Invoking into them.

If you're using the compat switch IncludeAllContentForSelfExtract, then you will replicate the 3.1 behavior, in which case AppContext.BaseDirectory should still be the extraction directory.

[agocke]

There actually may be a workaround if you're only extracting native libraries: AppContext.GetData("NATIVE_DLL_SEARCH_DIRECTORIES") should contain both the exe path and the native DLL extraction path, separated by a :.

@vitek-karas @elinor-fung is `NATIVE_DLL_SEARCH_DIRECTORIES a decent workaround for finding the native library extraction path if an app wants to delete its files after execution?

[vitek-karas]

AppContext.GetData("NATIVE_DLL_SEARCH_DIRECTORIES")

It's only sort of a workaround - that property can contain more than one path, in which case the app would have to reason about which one is the extraction path.

I'm sure that mirroring LoadLibrary from in C# is no issue nowadays and works fine if implemented correctly.

Duplicating all of LoadLibrary on Windows is.... really hard (regardless of which language I chose to do it in). Doing something which works for simple native .dlls might be possible in the basic sense (as @MichalStrehovsky pointed out, it would still break things like debuggers), but more complex native dlls... I would not even try to attempt that. For us it would be basically endless source of weird bugs - and lot of things simply would not work (not only debuggers, but for example native Win32 resources would not be loadable from such a library - because the Win32 APIs to access native resources would not know about the module).

We had to implement something "like" this to be able to load Ready2Run images directly from the bundle - Ready2Run images are basically normal native .dll files with added "IL" and normally they are loaded via LoadLibrary by the runtime. And even this was definitely not easy to do - and that's for native libraries which we know exactly how they look like and what they can and cannot contain.

Leaving everything but managed code next to exe will defeat whole point of using single file.

I understand that for some customers exactly one file is the only desired behavior. That said when we asked people why they want to use single-file, quite a few replied that they don't want ~200 files because it's hard to navigate, but having a few files would work for them. I do agree that calling the feature "single-file" while it's not always "single" is a bit weird.

As far as I know the only robust true single-file solution would be to use platform linker and require all native dependencies to be provided as object files or libs.

[tomrus88]

Sure, that NATIVE_DLL_SEARCH_DIRECTORIES can be used as dirty workaround, but things would have been much easier and clean if something like this #29456 (now closed) was added...

[agocke]

There were a number of different APIs there, and I think some don't make sense in light of the long-term plan to always embed managed assemblies and only extract native libraries, but an API for accessing the native DLL extraction path doesn't seem unreasaonable.

It's way too late to take this kind of change for 5.0, but I'd be happy to keep this or another issue around for tracking an API addition to find the native DLL extraction directory in 6.0.

[iSazonov]

there is NO reason to not implement loading native binaries from memory
Do you have pointers on how to do that? Windows doesn't have APIs for this. .NET 5 no longer extracts on Linux because there it's possible.

If it is not for 6.0 but for future why not ask Windows team to add this API?

[MichalStrehovsky]

If it is not for 6.0 but for future why not ask Windows team to add this API?

It will not work on Windows 7, the best operating system in the world (sharing the spot with Windows XP).

[agocke]

If it is not for 6.0 but for future why not ask Windows team to add this API?

We are in discussions, but loading shared libraries from memory is actually not possible even in POSIX. We would likely have to create a custom implementation for every operating system variant, including every Unix variant.

Surveying the options, it looks like almost everyone chooses to simply create temporary files for shared libraries. It just doesn't seem worth the trouble/complexity to do otherwise.

That said, if simple, reliable implementations were to be proposed, I think we'd be happy to take them.

[wegylexy]

For WPF, I've got a workaround. #59017

But if I want to build a library which may be used by non-WPF, I'll need a generic way to get the extraction directory.

[vitek-karas]

For WPF, I've got a workaround. #59017
But if I want to build a library which may be used by non-WPF, I'll need a generic way to get the extraction directory.

If you know that the process loaded a native library and which one it was (or you can make it do that), then you can use the same trick as in #59017 on any app, not just WPF.

Personally I don't think we should add a full public managed API for this (the API is very specific to one deployment option, and defining its behavior in all other cases would be weird). I can see us adding a new runtime property, so that apps could access this via something like AppContext.GetData("SINGLE_FILE_EXTRACTION_PATH") (or similar).

[reijerh]

Ran into this issue because our project distributes a native .exe file that we want to start in code via Process.

IncludeNativeLibrariesForSelfExtract does include this .exe, but it extracts it to a location I can't access from code.

I guess I have to use IncludeAllContentForSelfExtract, which is a pity because now everything from the build will be included and the managed libs will now be loaded from disk rather than from memory. Ah well.

[vitek-karas]

@reijerh did you consider storing the executable as a managed resource stream and writing it to disk yourself when necessary? It would also give you full control over where the executable lives on disk (The default extraction location for .NET is sometimes not ideal). It would also speed up startup of your app, since you would not need to extract it to disk unless really necessary, unlike the default extraction which happens on startup always.