Azure Key Vault: Could not load file or assembly System.Buffers

[jamers99]

Trying to get a secret from Azure Key Vault and I'm getting this exception on client.GetSecretAsync(string) in a .NET Framework 4.7.1 C# project.

Azure.Identity.AuthenticationFailedException
HResult=0x80131500
Message=ClientSecretCredential authentication failed.
Source=Azure.Identity
StackTrace:
at Azure.Identity.ClientSecretCredential.d__16.MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Azure.Security.KeyVault.ChallengeBasedAuthenticationPolicy.d__9.MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Azure.Security.KeyVault.ChallengeBasedAuthenticationPolicy.d__8.MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Threading.Tasks.ValueTask.ThrowIfCompletedUnsuccessfully()
at Azure.Core.Pipeline.RetryPolicy.d__11.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Azure.Core.Pipeline.RetryPolicy.d__11.MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.d__1.MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.d__1.MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Azure.Core.Pipeline.HttpPipeline.d__10.MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Azure.Security.KeyVault.KeyVaultPipeline.d__27.MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Azure.Security.KeyVault.KeyVaultPipeline.d__211.MoveNext() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Azure.Security.KeyVault.Secrets.SecretClient.<GetSecretAsync>d__8.MoveNext() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter1.GetResult()
at Common.AzureSettings.d__2.MoveNext() in C:\Builds\ebms-2XX\ebms-234-dev\ebms\EbmsServices\Common\AzureSettings.cs:line 43

Inner Exception 1:
AggregateException: Retry failed after 4 tries.

Inner Exception 2:
RequestFailedException: Error while copying content to a stream.

Inner Exception 3:
HttpRequestException: Error while copying content to a stream.

Inner Exception 4:
FileLoadException: Could not load file or assembly 'System.Buffers, Version=4.0.2.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)

Fusion log from the first try

=== Pre-bind state information ===
LOG: DisplayName = System.Buffers, Version=4.0.2.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51
(Fully-specified)
LOG: Appbase = file:///C:/Builds/ebms-2XX/ebms-234-dev/ebms/EBMSServerManager/Tests/EBMSServerManagerLibraryTests/bin/x86/Debug
LOG: Initial PrivatePath = NULL
Calling assembly : Azure.Core, Version=1.0.2.0, Culture=neutral, PublicKeyToken=92742159e12e44c8.
===
LOG: This bind starts in default load context.
LOG: Using application configuration file: C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Common7\IDE\Extensions\TestPlatform\testhost.x86.exe.Config
LOG: Using host configuration file:
LOG: Using machine configuration file from C:\Windows\Microsoft.NET\Framework\v4.0.30319\config\machine.config.
LOG: Post-policy reference: System.Buffers, Version=4.0.2.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51
LOG: Attempting download of new URL file:///C:/Builds/ebms-2XX/ebms-234-dev/ebms/EBMSServerManager/Tests/EBMSServerManagerLibraryTests/bin/x86/Debug/System.Buffers.DLL.
WRN: Comparing the assembly name resulted in the mismatch: Build Number
ERR: Failed to complete setup of assembly (hr = 0x80131040). Probing terminated.

[jamers99]

Here is a very simple unit test project that reproduces the issue, just run the test.
UnitTestProject1.zip

[jsquire]

//cc: @pakrym

[AlexGhiondea]

@jamers99 do you have a way to run the same project on .NET Framework 4.7.2 or higher? In 4.7.1 there are known issues with consuming .NET Standard assemblies and I wonder if you have run into those in this situation.

What version of Visual Studio are you using?

I am trying to reproduce this now.

[jamers99]

Hey @AlexGhiondea thanks for taking this so quickly! I'm trying to upgrade the test project to see if it makes a difference. I think we should be able to do that.

I'm using Visual Studio Enterprise 16.5.2.

[jamers99]

Same problem :( I cleaned and rebuilt.

[AlexGhiondea]

Thanks for the confirmation. I was able to repro this on 4.7.2 with VS 2019.

Here is another question -- can you use a .NET Core unit test project instead?

I think the issue is that the System.Buffers dependency is coming from a transitive dependency and because this is a unit test project I don't think there aren't any knobs that we can tweak to force a binding redirect to use the 4.0.3 version of the assembly.

@joperezr do you happen to have more insights here?

[jamers99]

@AlexGhiondea no we are not able to use .NET Core. I actually tested this originally in a .NET Core project and it worked nicely, but when i tried to integrate it into our .NET Framework service, it didn't work.

[AlexGhiondea]

Thanks @jamers99. Let's see if @joperezr has a suggestion for how to do this.

One thing you could try would be to introduce a binding redirect in the testhost's app.config that would redirect System.Buffers from 4.0.2.0 to 4.0.3.0. That way, the assemblies would unify to a single version and I expect you will be able to run the tests that way. Unfortunately this does have sideeffects (now every unit test will have that unification).

[joperezr]

While I haven't looked at the repro yet, this does in deed seem to be a problem with a missing binding redirect. When using test projects, it might be tricky as I believe that by default we only generate binding redirects for projects that produce *.exe, and not for dlls. To me this looks like the case where two different assemblies in your closure need System.Buffers, one needs the 4.0.2.0 version but another one depends on a newer package so it is bumping the dependency up to the 4.0.3 one which is the one that ends up in your ouptut folder. During execution, if you don't have a binding redirect telling the assembly that depends on 4.0.2 version that it should just load 4.0.3, you will hit the exception you are hitting. I believe testhost.x86 is one of those test hosts that will take into account dll.config files when loading assemblies, so would you mind adding the following to your projects in order to force msbuild to produce a dll.config for you?:

<PropertyGroup>
  <GenerateBindingRedirectsOutputType>true</GenerateBindingRedirectsOutputType>
</PropertyGroup>

[joperezr]

Here is another issue we are currently tracking which seems very similar to this one: dotnet/runtime#27774

[jamers99]

That worked!! Here is the config file it generated. Now to see if it will work in our real project...
UnitTestProject1.dll.zip

[jamers99]

Nice, that solution worked in the real project. Thanks so much!
But this wasn't a very intuitive experience 😬 and I don't think it should be a permanent solution. Will steps be taken to prevent the need for this in the future?

[joperezr]

Glad to hear that worked for you. I totally agree with this not being intuitive, and I would suggest logging an issue on the https://github.com/dotnet/sdk or the https://github.com/Microsoft/MSBuild repos in order to make it such that projects that don't have an .exe output type will also be considered (which is the case for test projects and also for some ASP.NET projects). They would really be the experts that could make this scenario better.