Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix native library loading on Linux #1732

Closed
wants to merge 2 commits into from
Closed

Fix native library loading on Linux #1732

wants to merge 2 commits into from

Conversation

qmfrederik
Copy link
Contributor

  • GetGlobalSettingsNativeLibraryPath is currently broken because it never adds the lib prefix to the native library name (libgit2 does not contain the lib prefix)
  • In ResolveDll, check whether the runtimes directory exists before listing child directories. In some configurations (e.g. NerdBank.GitVersioning) this directory may not exist, and the code would throw an exception.

@qmfrederik
Copy link
Contributor Author

qmfrederik commented Oct 28, 2019

/cc @tmds , this fixes a regression introduced by #1714

@tmds
Copy link
Contributor

tmds commented Oct 28, 2019

/cc @tmds , this fixes a regression introduced by #1732

By #1714.

I don't understand how though.

string nativeLibraryPath = Path.Combine(nativeLibraryDir, libgit2 + Platform.GetNativeLibraryExtension());

Isn't this a full path? Was something adding a prefix into this?

@qmfrederik
Copy link
Contributor Author

qmfrederik commented Oct 28, 2019

Path.Combine(nativeLibraryDir, libgit2 + Platform.GetNativeLibraryExtension()) would get you something/git2-ef5a385.so, but you need something/libgit2-ef5a385.so (the lib prefix is missing in the library name)

@qmfrederik
Copy link
Contributor Author

OK - your point is that string nativeLibraryPath = Path.Combine(nativeLibraryDir, libgit2 + Platform.GetNativeLibraryExtension()); was used previously, and your question is why that code is now considered broken by me.

I think that code has always been broken.

However, NerdBank.GitVersioning used a custom AssemblyLoadContext which loaded the native library (and added the lib prefix).

Because LibGit2Sharp now uses the NativeLibrary class, that no longer works and the underlying bug is exposed.

Does that make some sense?

@ethomson
Copy link
Member

GetGlobalSettingsNativeLibraryPath is currently broken because it never adds the lib prefix to the native library name (libgit2 does not contain the lib prefix)

Why is this a problem? There used to be code in mono, then .NET Core, to add the lib prefix on non-Windows systems. Is there a regression here? Is this only different because of the directory prefix?

@qmfrederik
Copy link
Contributor Author

qmfrederik commented Oct 28, 2019

@ethomson Yes, as far as I know, .NET Core will add the lib prefix (and the .so/.dll/.dylib suffix) only if you pass a filename. It will not do so if you include a directory prefix.

E.g. [DllImport("git2")] would find libgit2.so (if it is in a directory in LD_LIBRARY_PATH), but [DllImport("/tmp/git2")] would not find /tmp/libgit2.so.

Furthermore, #1714 changed the native library loading mechanism to use NativeLibrary.Load (which is a good thing), which doesn't do add a prefix/suffix either if you pass a directory-prefixed path.

You can see this in https://andrewarnott.visualstudio.com/OSS/_build/results?buildId=1903&view=logs&jobId=084ed6c3-25bc-526e-5414-75f5931b38aa, the code failed to load the native library (and went to the fallback path, which also failed)

@tmds
Copy link
Contributor

tmds commented Oct 28, 2019

Does that make some sense?

Yes, The lib insertion is something that happens in NerdBank.GitVersioning and is no longer occurring since the NativeLibrary class is used.

foreach (var runtimeFolder in Directory.GetDirectories(Path.Combine(assemblyDirectory, "runtimes"), $"*-{processorArchitecture}"))
var runtimesDirectory = Path.Combine(assemblyDirectory, "runtimes");

if(Directory.Exists(runtimesDirectory))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this change unrelated? Does something bad happen with Directory.GetDirectories when the directory doesn't exist?

Copy link
Contributor Author

@qmfrederik qmfrederik Oct 28, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, calling Directory.GetDirectories for a directory which doesn't exist throws an exception:

/home/vsts_azpcontainer/.nuget/packages/nerdbank.gitversioning/3.1.29-beta-g67e296ae86/build/Nerdbank.GitVersioning.targets(63,5): error MSB4018: The "Nerdbank.GitVersioning.Tasks.GetBuildVersion" task failed unexpectedly. [/__w/1/s/lib/lib.csproj]
/home/vsts_azpcontainer/.nuget/packages/nerdbank.gitversioning/3.1.29-beta-g67e296ae86/build/Nerdbank.GitVersioning.targets(63,5): error MSB4018: System.TypeInitializationException: The type initializer for 'LibGit2Sharp.Core.NativeMethods' threw an exception. [/__w/1/s/lib/lib.csproj]
/home/vsts_azpcontainer/.nuget/packages/nerdbank.gitversioning/3.1.29-beta-g67e296ae86/build/Nerdbank.GitVersioning.targets(63,5): error MSB4018:  ---> System.IO.DirectoryNotFoundException: Could not find a part of the path '/home/vsts_azpcontainer/.nuget/packages/nerdbank.gitversioning/3.1.29-beta-g67e296ae86/build/MSBuildCore/runtimes'. [/__w/1/s/lib/lib.csproj]
/home/vsts_azpcontainer/.nuget/packages/nerdbank.gitversioning/3.1.29-beta-g67e296ae86/build/Nerdbank.GitVersioning.targets(63,5): error MSB4018:    at System.IO.Enumeration.FileSystemEnumerator`1.CreateDirectoryHandle(String path, Boolean ignoreNotFound) [/__w/1/s/lib/lib.csproj]
/home/vsts_azpcontainer/.nuget/packages/nerdbank.gitversioning/3.1.29-beta-g67e296ae86/build/Nerdbank.GitVersioning.targets(63,5): error MSB4018:    at System.IO.Enumeration.FileSystemEnumerator`1.Init() [/__w/1/s/lib/lib.csproj]
/home/vsts_azpcontainer/.nuget/packages/nerdbank.gitversioning/3.1.29-beta-g67e296ae86/build/Nerdbank.GitVersioning.targets(63,5): error MSB4018:    at System.IO.Enumeration.FileSystemEnumerator`1..ctor(String directory, Boolean isNormalized, EnumerationOptions options) [/__w/1/s/lib/lib.csproj]
/home/vsts_azpcontainer/.nuget/packages/nerdbank.gitversioning/3.1.29-beta-g67e296ae86/build/Nerdbank.GitVersioning.targets(63,5): error MSB4018:    at System.IO.Enumeration.FileSystemEnumerable`1..ctor(String directory, FindTransform transform, EnumerationOptions options, Boolean isNormalized) [/__w/1/s/lib/lib.csproj]
/home/vsts_azpcontainer/.nuget/packages/nerdbank.gitversioning/3.1.29-beta-g67e296ae86/build/Nerdbank.GitVersioning.targets(63,5): error MSB4018:    at System.IO.Enumeration.FileSystemEnumerableFactory.UserDirectories(String directory, String expression, EnumerationOptions options) [/__w/1/s/lib/lib.csproj]
/home/vsts_azpcontainer/.nuget/packages/nerdbank.gitversioning/3.1.29-beta-g67e296ae86/build/Nerdbank.GitVersioning.targets(63,5): error MSB4018:    at System.IO.Directory.InternalEnumeratePaths(String path, String searchPattern, SearchTarget searchTarget, EnumerationOptions options) [/__w/1/s/lib/lib.csproj]
/home/vsts_azpcontainer/.nuget/packages/nerdbank.gitversioning/3.1.29-beta-g67e296ae86/build/Nerdbank.GitVersioning.targets(63,5): error MSB4018:    at LibGit2Sharp.Core.NativeMethods.ResolveDll(String libraryName, Assembly assembly, Nullable`1 searchPath) [/__w/1/s/lib/lib.csproj]
/home/vsts_azpcontainer/.nuget/packages/nerdbank.gitversioning/3.1.29-beta-g67e296ae86/build/Nerdbank.GitVersioning.targets(63,5): error MSB4018:    at System.Runtime.InteropServices.NativeLibrary.LoadLibraryCallbackStub(String libraryName, Assembly assembly, Boolean hasDllImportSearchPathFlags, UInt32 dllImportSearchPathFlags) [/__w/1/s/lib/lib.csproj]
/home/vsts_azpcontainer/.nuget/packages/nerdbank.gitversioning/3.1.29-beta-g67e296ae86/build/Nerdbank.GitVersioning.targets(63,5): error MSB4018:    at LibGit2Sharp.Core.NativeMethods.git_libgit2_init() [/__w/1/s/lib/lib.csproj]
/home/vsts_azpcontainer/.nuget/packages/nerdbank.gitversioning/3.1.29-beta-g67e296ae86/build/Nerdbank.GitVersioning.targets(63,5): error MSB4018:    at LibGit2Sharp.Core.NativeMethods.InitializeNativeLibrary() [/__w/1/s/lib/lib.csproj]
/home/vsts_azpcontainer/.nuget/packages/nerdbank.gitversioning/3.1.29-beta-g67e296ae86/build/Nerdbank.GitVersioning.targets(63,5): error MSB4018:    at LibGit2Sharp.Core.NativeMethods..cctor() [/__w/1/s/lib/lib.csproj]
/home/vsts_azpcontainer/.nuget/packages/nerdbank.gitversioning/3.1.29-beta-g67e296ae86/build/Nerdbank.GitVersioning.targets(63,5): error MSB4018:    --- End of inner exception stack trace --- [/__w/1/s/lib/lib.csproj]
/home/vsts_azpcontainer/.nuget/packages/nerdbank.gitversioning/3.1.29-beta-g67e296ae86/build/Nerdbank.GitVersioning.targets(63,5): error MSB4018:    at LibGit2Sharp.Core.NativeMethods.git_libgit2_opts(Int32 option, UInt32 level, String path) [/__w/1/s/lib/lib.csproj]
/home/vsts_azpcontainer/.nuget/packages/nerdbank.gitversioning/3.1.29-beta-g67e296ae86/build/Nerdbank.GitVersioning.targets(63,5): error MSB4018:    at LibGit2Sharp.Core.Proxy.git_libgit2_opts_set_search_path(ConfigurationLevel level, String path) [/__w/1/s/lib/lib.csproj]
/home/vsts_azpcontainer/.nuget/packages/nerdbank.gitversioning/3.1.29-beta-g67e296ae86/build/Nerdbank.GitVersioning.targets(63,5): error MSB4018:    at LibGit2Sharp.GlobalSettings.SetConfigSearchPaths(ConfigurationLevel level, String[] paths) [/__w/1/s/lib/lib.csproj]
/home/vsts_azpcontainer/.nuget/packages/nerdbank.gitversioning/3.1.29-beta-g67e296ae86/build/Nerdbank.GitVersioning.targets(63,5): error MSB4018:    at Nerdbank.GitVersioning.GitExtensions.OpenGitRepo(String pathUnderGitRepo, Boolean useDefaultConfigSearchPaths) [/__w/1/s/lib/lib.csproj]
/home/vsts_azpcontainer/.nuget/packages/nerdbank.gitversioning/3.1.29-beta-g67e296ae86/build/Nerdbank.GitVersioning.targets(63,5): error MSB4018:    at Nerdbank.GitVersioning.VersionOracle.Create(String projectDirectory, String gitRepoDirectory, ICloudBuild cloudBuild, Nullable`1 overrideBuildNumberOffset, String projectPathRelativeToGitRepoRoot) [/__w/1/s/lib/lib.csproj]
/home/vsts_azpcontainer/.nuget/packages/nerdbank.gitversioning/3.1.29-beta-g67e296ae86/build/Nerdbank.GitVersioning.targets(63,5): error MSB4018:    at Nerdbank.GitVersioning.Tasks.GetBuildVersion.ExecuteInner() [/__w/1/s/lib/lib.csproj]

I assumed the intent of this method was to return IntPtr.Zero instead of throwing.

@ethomson
Copy link
Member

@ethomson Yes, as far as I know, .NET Core will add the lib prefix (and the .so/.dll/.dylib suffix) only if you pass a filename. It will not do so if you include a directory prefix.

That makes sense. Thanks for the information.

I'm going to tag @bording in as well for 👀 since this is his expertise and not mine.

@tmds
Copy link
Contributor

tmds commented Oct 28, 2019

@qmfrederik are you setting GlobalSettings.NativeLibraryPath or does Nerdbank.GitVersioning do that. I'm curious what the use-case is.

@qmfrederik
Copy link
Contributor Author

@tmds I don't think NerdBank.GitVersioning used to set GlobalSettings.NativeLibraryPath, and instead relied on the AssemblyLoadContext to make sure the native library is resolved correctly.

With the recent change that approach no longer works, so I crafted a PR which sets GlobalSettings.NativeLibraryPath.

But that didn't work because the lib prefix was missing. (The first symptom was an exception from Directory.GetDirectories, which confused me for a while).

The corresponding PR in NB.GV is dotnet/Nerdbank.GitVersioning#400 .

@tmds
Copy link
Contributor

tmds commented Oct 28, 2019

I don't think NerdBank.GitVersioning used to set GlobalSettings.NativeLibraryPath

GlobalSettings.NativeLibraryPath is null by default unless you're using netfx:

if (netFX)
{
// For .NET Framework apps the dependencies are deployed to lib/win32/{architecture} directory
nativeLibraryDefaultPath = Path.Combine(GetExecutingAssemblyDirectory(), "lib", "win32");
}
else
{
nativeLibraryDefaultPath = null;
}

So I wonder how you are hitting this on non-Windows platforms. Isn't someone setting NativeLibraryPath?

@qmfrederik
Copy link
Contributor Author

NerdBank.GitVersioning is (amongst others) consumed as an MSBuild task.
That MSBuild task doesn't consume LibGit2Sharp as a NuGet package reference.
Instead, the LibGit2Sharp library is included in the NerdBank.GitVersioning NuGet package.

This means the native libraries are not in their "default" location (in the runtimes folder of the LibGit2Sharp` NuGet package).
As a result, NerdBank.GitVersioning needs to make sure LibGit2Sharp correctly loads the native libraries.

In the past, this was done with an AssemblyLoadContext. GlobalSettings.NativeLibraryPath was never set (AFAIK)
This no longer works, so I updated NerdBank.GitVersioning to set GlobalSettings.NativeLibraryPath instead.

That exposed the difference in how the NB.GV AssemblyLoadContext constructed the path to the native library vs. how GetGlobalSettingsNativeLibraryPath does that.

Perhaps @AArnott can expain it better, but this is how I understand it works 😄 .

At the end of the day, the native library name contains the lib prefix on Unix, so I think GetGlobalSettingsNativeLibraryPath should add it too, since it constructs a path which starts with a the GlobalSettings.NativeLibraryPath directory prefix.

@tmds
Copy link
Contributor

tmds commented Oct 28, 2019

Thanks for explaining @qmfrederik , it makes sense to me.

@bording
Copy link
Member

bording commented Oct 29, 2019

I should be able to take a closer look at this soon. I know there's some stuff in this area that needs to be cleaned up / removed after a previous PR made some changes that turned out to not work on non-Windows OSes.

@qmfrederik
Copy link
Contributor Author

@bording , gentle ping. Let me know if there's anything else I can do to help.

@bording
Copy link
Member

bording commented Nov 16, 2019

I've been holding off looking at this because I knew when I did I was going to find a huge mess.

I've started looking at it today, and unfortunately I was right.

As best as I can tell, your PR that prompted this one (dotnet/Nerdbank.GitVersioning#400) is likely based on a faulty premise: that GlobalSettings.NativeLibraryPath actually works.

The way NativeLibraryPathcurrently works in master was introduced in #1563. However, despite what the xml comments and the code says, it doesn't actually work on non-Windows OSes. You see how in the NativeMethods constructor where it attempts to use it to call LoadWindowsLibrary or LoadUnixLibrary to override the DllImport logic? It turns out that actually only works on Windows. When on any unix, that call is ignored and the DllImports use the original path at set at process start. That's because you cannot adjust the load path of a unix process after it has started.

Then on top of that, we now have the logic introduced in #1714 that only comes into play when LibGit2Sharp is being used on .NET Core 3.0. It's also testing to see if NativeLibraryPath has been set (which is reasonable given the previous state of the code), but it's using it in a different way, and I think there's a different set of expectations on it.

This leads to a very confusing mess of there being different ways of trying to load the native library, and the code is currently far more intertwined than it should be:

  • .NET Framework on Windows
  • .NET Core 2.x on Windows
  • .NET Core 2.x on non-Windows
  • .NET Core 3.x

Not to mention there is code in there to try and support mono as well, though I'm increasingly of the opinion that is no longer worth it.

So where does that leave us? To be honest, I'm going to need to spend more time sorting through all of this, but I'm pretty sure that setting NativeLibraryPath is never going to be something that you want to do.

The ultimate goal here is that as a library consumer, you don't have to worry about which underlying native library is required for the OS you're on, and with .NET Core 3.0, it looks like there are finally APIs that can make that possible.

That means if you care about using LibGit2Sharp from .NET Core 2.0 MSBuild, you're definitely going to have to have all the RID logic yourself in a custom AssemblyLoadContext.

The key thing to figure out is how to ensure that still works while also moving forward to the new stuff .NET Core 3.0.

@qmfrederik Which version of .NET Core were you running when you hit the problems with Nerdbank.GitVersioning? What lead you to thinking NativeLibraryPath was needed in your PR?

Until it's more clear to me exactly which of the myriad code paths are impacted, I'd like to hold off on making any more changes here.

@qmfrederik
Copy link
Contributor Author

@bording

Which version of .NET Core were you running when you hit the problems with Nerdbank.GitVersioning?

.NET 3.0. This build https://andrewarnott.visualstudio.com/OSS/_build/results?buildId=1950 shows that NB.GV is failing on Ubuntu 19.04:

 The "Nerdbank.GitVersioning.Tasks.GetBuildVersion" task failed unexpectedly. [/__w/1/s/lib/lib.csproj]
 System.TypeInitializationException: The type initializer for 'LibGit2Sharp.Core.NativeMethods' threw an exception. [/__w/1/s/lib/lib.csproj]
  ---> System.IO.DirectoryNotFoundException: Could not find a part of the path '/home/vsts_azpcontainer/.nuget/packages/nerdbank.gitversioning/3.1.30-beta-gcd42c70667/build/MSBuildCore/runtimes'. [/__w/1/s/lib/lib.csproj]
    at System.IO.Enumeration.FileSystemEnumerator`1.CreateDirectoryHandle(String path, Boolean ignoreNotFound) [/__w/1/s/lib/lib.csproj]
    at System.IO.Enumeration.FileSystemEnumerator`1.Init() [/__w/1/s/lib/lib.csproj]
    at System.IO.Enumeration.FileSystemEnumerator`1..ctor(String directory, Boolean isNormalized, EnumerationOptions options) [/__w/1/s/lib/lib.csproj]
    at System.IO.Enumeration.FileSystemEnumerable`1..ctor(String directory, FindTransform transform, EnumerationOptions options, Boolean isNormalized) [/__w/1/s/lib/lib.csproj]
    at System.IO.Enumeration.FileSystemEnumerableFactory.UserDirectories(String directory, String expression, EnumerationOptions options) [/__w/1/s/lib/lib.csproj]
    at System.IO.Directory.InternalEnumeratePaths(String path, String searchPattern, SearchTarget searchTarget, EnumerationOptions options) [/__w/1/s/lib/lib.csproj]
    at LibGit2Sharp.Core.NativeMethods.ResolveDll(String libraryName, Assembly assembly, Nullable`1 searchPath) [/__w/1/s/lib/lib.csproj]
    at System.Runtime.InteropServices.NativeLibrary.LoadLibraryCallbackStub(String libraryName, Assembly assembly, Boolean hasDllImportSearchPathFlags, UInt32 dllImportSearchPathFlags) [/__w/1/s/lib/lib.csproj]
    at LibGit2Sharp.Core.NativeMethods.git_libgit2_init() [/__w/1/s/lib/lib.csproj]
    at LibGit2Sharp.Core.NativeMethods.InitializeNativeLibrary() [/__w/1/s/lib/lib.csproj]
    at LibGit2Sharp.Core.NativeMethods..cctor() [/__w/1/s/lib/lib.csproj]
    --- End of inner exception stack trace --- [/__w/1/s/lib/lib.csproj]
    at LibGit2Sharp.Core.NativeMethods.git_libgit2_opts(Int32 option, UInt32 level, String path) [/__w/1/s/lib/lib.csproj]
    at LibGit2Sharp.Core.Proxy.git_libgit2_opts_set_search_path(ConfigurationLevel level, String path) [/__w/1/s/lib/lib.csproj]
    at LibGit2Sharp.GlobalSettings.SetConfigSearchPaths(ConfigurationLevel level, String[] paths) [/__w/1/s/lib/lib.csproj]
    at Nerdbank.GitVersioning.GitExtensions.OpenGitRepo(String pathUnderGitRepo, Boolean useDefaultConfigSearchPaths) [/__w/1/s/lib/lib.csproj]
    at Nerdbank.GitVersioning.VersionOracle.Create(String projectDirectory, String gitRepoDirectory, ICloudBuild cloudBuild, Nullable`1 overrideBuildNumberOffset, String projectPathRelativeToGitRepoRoot) [/__w/1/s/lib/lib.csproj]
    at Nerdbank.GitVersioning.Tasks.GetBuildVersion.ExecuteInner() [/__w/1/s/lib/lib.csproj]

What lead you to thinking NativeLibraryPath was needed in your PR?

Couple of things:

  1. NB.GV does use a custom AssemblyLoadContext. The AssemblyLoadContext is not being invoked.
    a. I now assume this is because the NativeLibrary mechanism gets priority ("This per-assembly resolver is the first attempt to resolve native library loads initiated by this assembly."), throws an exception, and causes the native library loading mechanism to abort before the AssemblyLoadContext is invoked.
  2. This exception caused me to look at the new library loading mechanism in LibGit2Sharp, which includes this line:
    // Use GlobalSettings.NativeLibraryPath when set.
    a. I assumed this was the "new way to go", but the code path below contains a simple oversight, in that it forgets to add the lib prefix to the filename of the native library it tries to load.
    b. I actually validated this fix on an Ubuntu 19.04 box. It works.
  3. Based on your feedback, I think it's sufficient to just patch ResolveDll so that it does not throw if runtimesDirectory does not exit. In that case, the AssemblyLoadContext still gets a chance to load the native library, and all is well.
  4. I still think this PR is correct and fixes a scenario which is broken, but I've also spent enough time fighting the .NET Core native library loading mechanisms to understand you want to limit the amount of change here.

Net, I'll create a new PR which focuses on ResolveDll not throwing an exception when not needed, and letting the AssemblyLoadContext do its job instead.

Does that make sense?

@tmds
Copy link
Contributor

tmds commented Nov 20, 2019

Based on your feedback, I think it's sufficient to just patch ResolveDll so that it does not throw if runtimesDirectory does not exit. In that case, the AssemblyLoadContext still gets a chance to load the native library, and all is well.

Aha! This is the missing piece in the puzzle. I was wondering why the AssemblyLoadContext was no longer being used.

Net, I'll create a new PR which focuses on ResolveDll not throwing an exception when not needed, and letting the AssemblyLoadContext do its job instead.
Does that make sense?

If not throwing causes the AssemblyLoadContext to be used, that makes sense to me.
This will make Nerdbank.GitVersioning work as before.
Making GlobalSettings.NativeLibraryPath on Linux can be considered a different issue.

@tmds
Copy link
Contributor

tmds commented Nov 20, 2019

If not throwing causes the AssemblyLoadContext to be used, that makes sense to me.

This is tackled in: #1741

@qmfrederik
Copy link
Contributor Author

Thanks @tmds !

@bording
Copy link
Member

bording commented Nov 20, 2019

@qmfrederik Given that #1741 is merged, I assume it's okay to close this one?

@qmfrederik
Copy link
Contributor Author

@bording Up to you. My scenario works, so I don't really need this PR anymore. On the other hand, I still think GetGlobalSettingsNativeLibraryPath is broken by not adding the lib prefix on Unix, something which this PR addresses.

Feel free to close the PR if other concerns (like the ones you mentioned above) need to be addressed first.

@bording
Copy link
Member

bording commented Nov 20, 2019

Given that NativeLibraryPath won't work on unix even if it had the lib prefix in the path, let's go ahead and close this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants