Compilation libraries cannot be found error when hosting .NET core in Nano Server containers

[HuanhuanSunMSFT]

Here is a Repro Project.

After launch the ASP.NET core web app, browse the website with IP address will return 500 error. Here is the error within the nano server container.

PS C:\app> dotnet --additionalprobingpath "c:\.nuget\packages" "bin\Debug\netcoreapp1.1\myapp.dll"
Hosting environment: Production
Content root path: C:\app
Now listening on: http://+:80
Application started. Press Ctrl+C to shut down.
fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[0]
      An unhandled exception has occurred: Can not find compilation library location for package 'microsoft.applicationinsights'
System.InvalidOperationException: Can not find compilation library location for package 'microsoft.applicationinsights'
   at Microsoft.Extensions.DependencyModel.CompilationLibrary.ResolveReferencePaths()
   at System.Linq.Enumerable.<SelectManyIterator>d__159`2.MoveNext()
   at Microsoft.AspNetCore.Mvc.Razor.Compilation.MetadataReferenceFeatureProvider.PopulateFeature(IEnumerable`1 parts, MetadataReferenceFeature feature)
   at Microsoft.AspNetCore.Mvc.ApplicationParts.ApplicationPartManager.PopulateFeature[TFeature](TFeature feature)
   at Microsoft.AspNetCore.Mvc.Razor.Internal.RazorReferenceManager.GetCompilationReferences()
   at System.Threading.LazyInitializer.EnsureInitializedCore[T](T& target, Boolean& initialized, Object& syncLock, Func`1 valueFactory)
   at Microsoft.AspNetCore.Mvc.Razor.Internal.RazorReferenceManager.get_CompilationReferences()
   at Microsoft.AspNetCore.Mvc.Razor.Internal.DefaultRoslynCompilationService.CreateCompilation(String compilationContent, String assemblyName)
   at Microsoft.AspNetCore.Mvc.Razor.Internal.DefaultRoslynCompilationService.Compile(RelativeFileInfo fileInfo, String compilationContent)
   at Microsoft.AspNetCore.Mvc.Razor.Internal.RazorCompilationService.Compile(RelativeFileInfo file)
   at Microsoft.AspNetCore.Mvc.Razor.Internal.CompilerCache.CreateCacheEntry(String relativePath, String normalizedPath, Func`2 compile)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Mvc.Razor.Internal.CompilerCache.GetOrAdd(String relativePath, Func`2 compile)
  at Microsoft.AspNetCore.Mvc.Razor.Internal.DefaultRazorPageFactoryProvider.CreateFactory(String relativePath)
   at Microsoft.AspNetCore.Mvc.Razor.RazorViewEngine.CreateCacheResult(HashSet`1 expirationTokens, String relativePath, Boolean isMainPage)
   at Microsoft.AspNetCore.Mvc.Razor.RazorViewEngine.OnCacheMiss(ViewLocationExpanderContext expanderContext, ViewLocationCacheKey cacheKey)
   at Microsoft.AspNetCore.Mvc.Razor.RazorViewEngine.LocatePageFromViewLocations(ActionContext actionContext, String pageName, Boolean isMainPage)
   at Microsoft.AspNetCore.Mvc.Razor.RazorViewEngine.FindView(ActionContext context, String viewName, Boolean isMainPage)
   at Microsoft.AspNetCore.Mvc.ViewEngines.CompositeViewEngine.FindView(ActionContext context, String viewName, Boolean isMainPage)
   at Microsoft.AspNetCore.Mvc.ViewFeatures.Internal.ViewResultExecutor.FindView(ActionContext actionContext, ViewResult viewResult)
   at Microsoft.AspNetCore.Mvc.ViewResult.<ExecuteResultAsync>d__26.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeResultAsync>d__30.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeNextResultFilterAsync>d__28.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow(ResultExecutedContext context)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeNextResourceFilter>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow(ResourceExecutedContext context)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeAsync>d__20.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Builder.RouterMiddleware.<Invoke>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>d__6.MoveNext()
fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[0]
      An exception was thrown attempting to execute the error handler.
System.ArgumentException: An item with the same key has already been added. Key: System.Object
   at System.ThrowHelper.ThrowAddingDuplicateWithKeyArgumentException(Object key)
   at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeNextResourceFilter>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow(ResourceExecutedContext context)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeAsync>d__20.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Builder.RouterMiddleware.<Invoke>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>d__6.MoveNext()
fail: Microsoft.AspNetCore.Server.Kestrel[13]
      Connection id "0HL2I15H5I3IV": An unhandled exception was thrown by the application.

[srivatsn]

Is this an issue where preservecompilationcontext is not preserving enough? @dsplaisted can you take a look?

[dsplaisted]

It looks like the --additionalprobingpath parameter isn't making its way to the dependency context that Razor is using to resolve references at runtime.

@eerhardt @gkhanna79 Is there a way to flow this information through?

[dsplaisted]

Also ping @pranavkm and @natemcmaster on whether Razor compilation can respect the --additionalprobingpath.

[pranavkm]

@dsplaisted all we do in Mvc is call DependencyContext.Load(<launched application assembly>). It doesn't look like https://github.com/dotnet/core-setup/blob/master/src/Microsoft.Extensions.DependencyModel/DependencyContext.cs necessarily understands the --additionalprobingpath option. Is that available to the application at runtime? Perhaps we could make the API consume it.

[gkhanna79]

@pakrym should chime in on this as he owns that code.

[natemcmaster]

@HuanhuanSunMSFT I downloaded your app and it builds/runs fine outside of the container. I'm guessing the container setup has something to do with this error. I noticed your dockerfile is using the "runtime" Docker image. Normally you would use this image by executing dotnet publish first, and then copying the published output into the Docker image. The publish step is important as it will gather all of your dependencies into the right place. See https://github.com/dotnet/dotnet-docker#run-a-net-core-application-with-the-net-core-runtime-image for more details.

[eerhardt]

The DependencyModel's resolution logic only knows how to look in a few specific places. None of them are in the --additionalprobingpaths at the moment. (As far as I can tell, the "dotnet.exe" host doesn't pass these probing paths through to managed code, it only consumes them itself in native code.)

Here are the places DependencyModel knows to look:
https://github.com/dotnet/core-setup/blob/eca152525e19a41d489cd5845afa14982653cc0a/src/Microsoft.Extensions.DependencyModel/Resolution/PackageCompilationAssemblyResolver.cs#L41-L64

Basically:

1. NUGET_PACKAGES env variable
2. %USERPROFILE%\.nuget\packages

There are other places that are probed by DependencyModel, but the above 2 are the ones that are relevant to your scenario. So if you really want your scenario to work, as is, you would need to map one of those 2 places inside your docker container.

All that being said, I agree with @natemcmaster above - using dotnet publish is the correct way to get an app from a "dev machine" to another machine where you intend to run it.

[dsplaisted]

I've reproduced this issue outside of a docker container (by renaming the NuGet packages folder), and verified that setting the NUGET_PACKAGES environment variable fixes the issue both outside and inside a docker container.

You have to specify both the --additionalprobingpaths parameter to dotnet as well as setting the NUGET_PACKAGES environment variable. The host only looks at the parameter, while the Razor compilation only looks at the environment variable.

So @HuanhuanSunMSFT, I think for now the fix to this issue is to set the NUGET_PACKAGES environment variable in the container configuration, as well as the additional probing paths parameter.

It would be good to have a unified way of setting the path to NuGet packages that works for both the host and Razor compilation, but that's not as urgent so I'm going to move this out of 1.0 RTM.

[dazhao-msft]

@HuanhuanSunMSFT let's add NUGET_PACKAGES to docker-compose.vs.debug.yml for Nano scenarios.

[HuanhuanSunMSFT]

Thanks everyone. I'll take @dsplaisted suggestion to set nuget packages path as an environment variable.
@natemcmaster The reason we need volume mapping instead of publishing and packaging to container is performance. For debugging scenario, we want the process as simplified as possible to reduce the time between F5 clicked and breakpoints hit.

[eerhardt]

FYI - the underlying issue here of --additionalprobingpaths not being passed to the DependencyModel is being fixed in 2.0 with dotnet/core-setup#2039.