[Verify/Document] Native (conditional) dependency support.

[lilith]

Originally posted on Paket, but since this is something NuGet is considering as well, I'm adding it here so we can track and (hopefully sync efforts). I hear that the ASP.NET team is doing something with native dependencies, but I can't find a spec for it. I suspect that a lockfile and transitive dependency support are pre-requisites for this.

A lot has been written about this (challenging) topic; so I'll link to what I've found (and please add more links in the comments).

• https://stackoverflow.com/questions/19478775/add-native-files-from-nuget-package-to-project-output-directory
• Support consuming and producing nuget packages with native binaries aspnet/dnx#402
• https://nuget.codeplex.com/workitem/1221
• https://nuget.codeplex.com/workitem/679
• https://nuget.codeplex.com/discussions/412012
• http://ilnumerics.net/blog/anycpu-computing-limping-platform-specific-targets-and-a-happy-deployment-end/
• http://youku.io/questions/392214/nuget-powershell-how-to-add-native-dependencies-how-to-add-files-to-a-project
• https://nuget.codeplex.com/discussions/446656
• Sort out SQLite installer for winrt MvvmCross/MvvmCross#307
• https://nuget.codeplex.com/discussions/286521
• https://nuget.codeplex.com/discussions/230521
• https://nuget.codeplex.com/workitem/109

I've been using a https download-during-boot approach for ImageResizer, but that is slow, unreliable, and annoying. I tried to create an example of how to build the ideal native/managed hybrid project, failed, then started a project to try to hot-fix the problem at runtime, and hit another series of roadblocks.

I've identified a few invalid assumptions that seem responsible for the current state of things.

Some of these are somewhat comical considering how easy it is to parse binaries for the major platforms and determine runtime compatibility.

1. It is bad to assume all managed dlls are AnyCPU. x86 and x64-only binaries continue to have an important place. a) C++/CLI is still important, but only targets Win32 and x64. b) Binding generation tools like CppSharp cannot produce AnyCPU C# - the structure layouts (and other details) are calculated based on a pointer width assumption. c) There's also manually written C# that uses unsafe code or performs P/Invokes, and doesn't use IntPtr in all the right places, and is therefore x86 or x64-only.
2. It is bad to assume that precompiled native binaries are appropriate for every operating system distribution. Source compilation needs to be first-class if we want to target a large number of linux or bsd distributions; we can't precompile for everything.
3. It is bad to assume that binaries are small enough that all the permutations can go in the same zipped nuget package. OpenCV base libraries could easily be > 2 gigabytes if you did this. You'd also exhaust your server's disk space. And likely your bandwidth allotment. And requests would time out.
4. It is bad to assume that a windows machine is capable of compiling native code.

---

Removing these assumptions, what new requirements are we left with?

1. Nuget packages need to be able to reference other nuget packages - conditionally - based on architecture and operating system. This likely means that we need conditions in the lockfile, too - use version X of package Y on platform Z, etc. At build time, the right versions are copied.
2. Nuget packages need to be able to describe native binaries - or rather - arbitrary files, and provide multiple versions for each target platform. For small binaries, a common pattern is likely to combine these two approaches, and provide x86/x64 binaries for windows, and a conditional reference for other platforms. For larger binaries, it's likely that the 'main' package will be empty, and simply list conditional, plat-specific packages as dependencies.
3. Build time. This is a bit harder. What needs to happen?
   a) We need to gather the referenced files (nuget references, mind you), and verify that the output folder does not have any conflicting named files. If there are conflicting names, the output folder version MUST be deleted, so that we can AssemblyResolve or LoadLibrary the correct version. We then copy each of the files to an appropriate subfolder of the output folder (or, if AnyCPU, the output folder itself). Since VisualStudio is blind to compatibility (by choice, one must assume), we may end up fighting with the build process a bit. Perhaps disabling copylocal? Another nice sanity check would be to simply parse the binary headers of everything in the output folder and ensure they are all able to run on a common environment.
4. Run time. This is where developers have to do a little magic, and call AssemblyResolve for non-AnyCPU managed dlls, and LoadLibrary (or a platform alternative) for the native dependencies. Given a standardized convention, we can make this co-operation possible without .NET runtime changes (although not optimal, since Assembly.Load won't use the default security context). It would be much better if .NET would modify the search path based on platform, as it does for culture. It would also be great if ASP.NET would apply some intelligence or header parsing to assemblies before globbing them all into memory. It's not hard, and only requires between 1 and 3 I/O reads of a few hundred bytes.
5. Tooling. Tooling needs to understand that there are non-.NET dependencies involved. Unit test runners, in particular, are known for leaving behind native dependencies when they copy (or shadow copy) assembles for testing. These will need to understand the (transitive) dependencies. Perhaps we should emit some kind of manifest? While we can piggy-back on .NET assemblies to document dependencies (via resource manifests or regular assembly attributes), those .NET assemblies would need to document the final transitive set of native dependencies, which might be difficult to achieve.

So, I guess

• Step 1: Establish (a) plat/arch conditions, and (b) target strings we can use as subfolders.
• Step 2: Describe required changes to the nuget package specification
• Step 3: Implement handling in paket for nuget conditions and native file manifests, through to lockfile.
• Step 4: Implement paket support for triggering bash/.bat build scripts (perhaps requiring user authorization on a per-commit-id basis, if applied to a git reference)? CMake is probably most likely to be used, but if we require Git, we can require Bash.
• Step 5: Implement build-time support for copying and manifest generation.
• Step 6: Submit PRs to major runners and web frameworks to handle manifests and assembly loading properly.
• Step 7: Make .NET into a real platform, where C interop is practical, so we can play with the big kids.

Conditions:

• pointerSize=32|64. Let's say that our MSIL makes an assumption about pointer size, but not platform.
• endianess="little|big" Sometimes this matters more than architecture, and ARM can switch between endianess modes.
• architecture=x86|x64|IA64|Alpha|MIPS|HPPA|PowerPC|SPARC32|s390|s390x - Should probably support everything Mono does
• os="posix|winish|linux|osx|win7|win8|win10" I'm not sure how to best divide windows (or linux) operating systems into groups, or to number them for easy inequality testing, but we probably want to establish sane identifiers that correspond to common build/api compat targets.

Given that a fallback mode (building from source) is likely popular, we want to make it easy to ensure that only 1 reference from a conditional set is chosen. We should probably group them within another element or provide an id that prevents duplicates.

Target strings

Target string need to be as generic as can be permitted based on their restrictions.

/ - root is AnyCPU
/32b/ - managed, assumes 32-bit pointer, otherwise portable
/64b/ - managed, assumes 64-bit pointer, otherwise portable
/x86/winish/ - 32-bit, requires windows APIs.

pointer size, architecutre, and endianess are combined into the first string. Pointer size and endianess are only included if the architecture string doesn't make them redundant. I.e, we would see /ARM-little/ and /ARM-big/, but not /x86-little/.

[davidfowl]

@nathanaeljones Have you done any case studies of other package managers that may have these same problems? Do they do conditional dependencies based on arbitrary pivots? Does that condition on the dependency edge affect version selection?

I know npm supports native modules being compiled on the target machine at install time. Are you suggesting that we support the full spectrum of things?

a) We need to gather the referenced files (nuget references, mind you), and verify that the output folder does not have any conflicting named files. If there are conflicting names, the output folder version MUST be deleted, so that we can AssemblyResolve or LoadLibrary the correct version. We then copy each of the files to an appropriate subfolder of the output folder (or, if AnyCPU, the output folder itself). Since VisualStudio is blind to compatibility (by choice, one must assume), we may end up fighting with the build process a bit. Perhaps disabling copylocal? Another nice sanity check would be to simply parse the binary headers of everything in the output folder and ensure they are all able to run on a common environment.

We're enhancing MSBuild in vs14 with NuGet aware targets that understand runtime/vs compile time.

Run time. This is where developers have to do a little magic, and call AssemblyResolve for non-AnyCPU managed dlls, and LoadLibrary (or a platform alternative) for the native dependencies. Given a standardized convention, we can make this co-operation possible without .NET runtime changes (although not optimal, since Assembly.Load won't use the default security context). It would be much better if .NET would modify the search path based on platform, as it does for culture. It would also be great if ASP.NET would apply some intelligence or header parsing to assemblies before globbing them all into memory. It's not hard, and only requires between 1 and 3 I/O reads of a few hundred bytes.

This is what we do today with things like kestrel in ASP.NET 5. See my comment here imazen/Imazen.NativeDependencyManager#5 (comment). I'm not sure about making these new things work down level on all CLRs but newer ones might be more viable.

Tooling. Tooling needs to understand that there are non-.NET dependencies involved. Unit test runners, in particular, are known for leaving behind native dependencies when they copy (or shadow copy) assembles for testing. These will need to understand the (transitive) dependencies. Perhaps we should emit some kind of manifest? While we can piggy-back on .NET assemblies to document dependencies (via resource manifests or regular assembly attributes), those .NET assemblies would need to document the final transitive set of native dependencies, which might be difficult to achieve.

Transitive dependencies are going to be understood with build time and runtime nuget resolution coming into the platform.

I'm actually not a fan of complicating the graph walk by adding arbitrary pivots to it. I think that's actually an intractable problem. I do however agree with the fact that we do need conventions to describe these things in NuGet packages and have appropriate behaviors at runtime and build time.

[lilith]

Conditional packages based on platform seems to be very widespread. npm has optionaldependencies and platform filtering, bundler has :platform=> .

The problem with Bundler, (and the reason windows support is troubled) is its lack of support for pivots within the lockfile. Bundler supports platform pivots, but doesn't generate different lockfiles for each target. I think this is considered a flaw by the Bundler team (although I can't find the link; could have been mentioned at rubyconf 2014).

There are the choices I see:

1. Build from source.
2. Don't do anything operating-system specific, never use native code.
3. Have pivots.
4. Have both pivots and permit building from source.

I'm a fan of #4. When it's small, building from source is fine. But msvc can be 10-50x slower than gcc; building a tiny graphics library (libgd) with 3 dependencies takes over 45 minutes on windows. On linux, that's 4 minutes. Both options should be present.

I'm also not suggesting arbitrary pivots; rather just 4 pivots. That's a manageable number, I think. If you limit it to modern platforms, there are only 12 permutations. In practice, we're going to see binaries for windows, and source code for everything else. That's 3 permutations, 4-5 if you include ARM processors. I'd expect binaries for *nix platforms only where compilation is excessively slow.

Is the graph walk so resource-intensive that we can't take a 3-5x hit for the portions of that graph which have conditional-based native dependencies?

I'm not sure I see how the graph is an intractable problem, especially if we only have to "solve" it for the targets that the consuming project defines. Hey, — if the nuget server can send back dependency information for a package without sending the whole package — even the most naive approach should be just fine.

You should really talk to @wycats. He's the library package manager design expert, and he's dealt with this problem in many forms.

[davidfowl]

Walking the graph isn't hard, picking versions based on compatible pivots is hard and leads to unpredictability in the algorithm, which I think is much worse.

[lilith]

I would not select versions based on any of these pivots. I'm not even fully convinced that picking versions based on the .NET framework version is a good idea.

[lilith]

It's far better to "let it fail" than to make the algorithm unpredictable. If it fails, it can be fixed. If nothing fails, there is no pain point to repair, and that platform may remain stuck in the past forever.

[yishaigalatzer]

We are going to do something for RTM for sure. Will loop back when there is more meaty code to talk about.

[lilith]

I hope this can be designed in the open.

[lilith]

Are there any specs/documents/drafts we can review?

[yishaigalatzer]

Not yet, still working on it

[yishaigalatzer]

@nathanaeljones we are going to have project.json and a lock file as well as transitive restore supporting install.ps1 uninstall.ps1 content and targets fully in Visual Studio. The code is already public in NuGet but we are still playing around with it. It requires an msbuild task/target file that is being worked on as well (but is not public yet).

I don't understand the native picture 100% so I don't want to add misleading information until its properly documented.

[lilith]

@yishaigalatzer Have you resolved the issues with the msbuild team? Last I heard they were refusing to support NuGet (or any package manager, for that matter). I think the requirement was to enable reloading of the build file once the package manager has finished its work. Packet and NuGet both ran into the same issue, I think.

We'd all like to see a solution that can work without a VS-specific extension, but right now it appears to be at an impasse.

---

The other issue is that AppDomainSetup ShadowCopy functionality needs to be aware of that transitive list. xUnit's usage here. We've exhaustively explored workarounds to hack this in lieu of a .NET 4.6 feature addition, and the smallest change we can come up with is to add a read-only field to Assembly that provides the original (pre-shadow-copy) path of the assembly. Without that, there's no way to guess where the assembly came from, and where the missing files could be located.

Ideally, of course, the runtime would be aware of this lockfile and handle the xplat differences for us.

---

More detail about install/uninstall would be great. First-class native binary nuget packages would be helpful from a security point of view; wrappers wouldn't need to update for a security patch to be applied.

[yishaigalatzer]

There is no msbuild involvement. We run nuget restore before build in nuget 3. Try it out in the rc bits.

I'll read about the other issues you mentioned.

Paket is honestly a very limited view on what NuGet packages contain. And I can't speak to what its limitations are.

I'll make sure that once command line bits are available we will reach out to you with more details.

[yishaigalatzer]

We are now building NuGet based on lock files and project.json (as you can see in all our client repos). The release is going to be a bit after RTM, but the bits are already somewhat working.

We are not currently supporting pivots. But we are starting to finally working on documenting the process of what we got so far.

[yishaigalatzer]

So I don't think this is going to get fully resolved in the first release. I'm still leaving it at that milestone so we can keep it in the front, and turn this issue in a feedback/next steps kind of issues on top of that release

[yishaigalatzer]

@nathanaeljones here is a link to a package that addresses the native scenario -
https://www.myget.org/F/dotnet-core/api/v2/package/System.IO.Compression/4.0.0-beta-22927

[ghost]

👍 for having npm inspired optionalDependency, which do not get installed when certain CLI flag is not present during the installation. The program is supposedly work fine without optional dependencies. Think of it as MEF architecture, where if the binary is not present, some feature (say a menu item) will be absent, but the program as a whole will still function.

But

npm has optional dependencies and platform filtering

npm does not select native binary based on the os+architecture. Do you think it creates a package dynamically everytime you run npm install and make decisions which binary to pack? No. All packages using node-gyp for C++ binary dependency have this issue. From the link you provided, optionalDepenedency is something misleading in this regard. The os key is a flat filter, it just rejects the entire installation on non-permitted OS.
Last time I checked, this is one of the major challenge for modules using C++ dependencies, as the 'package' itself is a dumb / distilled form of your product and is essentially environment-agnostic.

Then how do they know which binary to keep?

• Compile on install (require python, C++ compiler, node-gyp). Not a very successful option for Windows users.
• Download the binary during the installation using custom install script
• Package all binaries within the package (some clean up after installation <- just being pendantic).

The decision whether the binary is compatible maybe complicated than just matching the architecture. node.js depends on v8, which has its own versions and then there are module versions etc. (node -p process.versions there are 5-6 constituents and a slight version change can and does break the native dependencies. There is no universal way of distributing native binary in node.js world and it is still a BIG challenge).

-> Spoken from series of personal painful experiences with npm/C++/node-gyp.

[yishaigalatzer]

Pushing a package that will only work on Paket is definitely a worse approach than making something that works everywhere. Yes it is harder today, and perhaps the answer is to eventually port nuget 3 down to 2013. The answer today is to use targets for down level visual studio support like win2D package does.

And as for implodes on any real-world project, I suggest filing issues, with as much detail as possible. And work through them together. The general statement above doesn't get us anywhere.

[dead-claudia]

NPM mostly delegates the step to node-gyp, which then handles OS dependencies and compiles the code itself using the system's compiler, unless a "preinstall" is specified in "scripts", which can itself call this. Frequently, packages override this to first check for an OS-specific precompiled binary and download it if it exists. The PhantomJS runtime and node-fibers do this.

PyPI requires everything to be prebuilt, and uses setuptools to simplify and abstract the process.

I think NuGet could support something similar.

[ghost]

@IMPinball, like @nathanaeljones noted:

It is bad to assume that a windows machine is capable of compiling native code.

Here is a shining example: nodejs/node-gyp#629 with a glimpse of hope (in the form of Build Tools 2015 without VS).

In npm's case, it is assumed that all Unix-like operating system are packed with C/C++ compilers, while on Windows only MSBuild is supported by node-gyp. The approach you mentioned about phantomJS (and node-sass) is good that prepare the binaries for as many OSs and arch as you can and for the remaining fallback to default node-gyp's compiler from source at installation time behavior, but at the same time, too much work for the maintainer to verify and test binaries on all possible OS and architectures.

In my experience, most people are happy if they just do npm install package and get things rolling, preferably without compilation step. Even on Unix-likes, npm+node-gyp's advertised approach to compile on the fly fails when we use relatively new feature-set of native languages which some Linux distribution's version's out of the box compilers might not support (CentOS-5.11 which many hosting environments are running till date packs with gcc 4.4, which does not even support nullptr!). So every maintainer of the package (should) have to mandate the minimum C/C++ compilers' version on target as well, which does not inspire many consumers..

NuGet may experience a totally different trend of packaging stuff if this ongoing relevant discussion in CoreFX repo: https://github.com/dotnet/corefx/issues/2302#issuecomment-156132918 get popular.

[dead-claudia]

@jasonwilliams200OK Sorry to burst your bubble, but I don't think that idea on npm is getting very many places, as great as it sounds...

[dead-claudia]

Although I could see the benefits of .NET Core's approach.

[ghost]

@IMPinball, i think you totally missed the point... but this is more or less what i am trying to say..

[dead-claudia]

I'm not immediately invested in this, but this is the one thing that's stopping me from learning the platform where I use Linux (and there's a lot of .NET jobs where I live).

[lilith]

I am very glad to see documented Runtime IDs have landed.

I do not see that <group> offers pivots by runtime ID. Is there a way to conditionally depend on a package based on runtime ID?

As discussed, it would be unexpected for the NuGet client to select a version based on what runtimes are supported by specific versions - rather, I would expect that the latest matching version number is used (based on semver evaluation), and no 'compatibility' checking is employed when selecting dependencies. If a runtime is missing for a specific version, restore fails, and the user must add a version specification. This ensures that there is no computational complexity in flattening transitive dependencies into a lockfile despite the presence of an additional runtime pivot.

Separating native dependencies into packages by runtime enables a very easy deployment scenario Every Travis and AppVeyor task is only responsible for uploading their own runtime's package. The last task can upload an aggregate package which references them, after testing they are all retrievable (with a retry loop).

[jainaashish]

Its been an year without any update here and not sure what was the last status since this is very long thread. So @nathanaeljones can you please check you if this still applicable for `PackageReference'? and if it does, then please open a new issue with relevant details so that we can take that forward.