[Bug Report] Flaky TLS authentication error with EdgeHub only on dotnet 6

[and-rewsmith]

Context

• OS, version, SKU and CPU architecture used: Ubuntu 18.04
• Application's .NET Target Framework : net6.0
• Device: Azure VM
• SDK version used: 1.36.4

Description of the issue

Happens ~5% of the time. Tried to obtain local reproduction with console app but it would not reproduce.

Same source code that exhibits this issue as previous issues:
#2351
#2353

We try to create a leaf device connecting to EdgeHub and sometimes we get a TLS authentication error. This did not happen before updating to dotnet 6. Exact error seems that it receives an unexpected message.

What are the next steps here? Assuming we should find a way to grab a wireshark log of TLS auth failure and attach it? Anything else I can do?

Our E2E framework test stack:

[17:13:28 INF] [ +96.398s] Completed test 'AuthorizationPolicyUpdateTest'
[17:13:34 INF] [  +5.167s] Completed test teardown
  Failed AuthorizationPolicyUpdateTest [1 m 52 s]
  Error Message:
     Expected: <Microsoft.Azure.Devices.Client.Exceptions.UnauthorizedException>
  But was:  <System.Security.Authentication.AuthenticationException: TLS authentication error.
 ---> System.AggregateException: One or more errors occurred. (Authentication failed, see inner exception.)
 ---> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception.
 ---> Interop+OpenSsl+SslException: SSL Handshake failed with OpenSSL error - SSL_ERROR_SSL.
 ---> Interop+Crypto+OpenSslCryptographicException: error:141A10F4:SSL routines:ossl_statem_client_read_transition:unexpected message
   --- End of inner exception stack trace ---
   at Interop.OpenSsl.DoSslHandshake(SafeSslHandle context, ReadOnlySpan`1 input, Byte[]& sendBuf, Int32& sendCount)
   at System.Net.Security.SslStreamPal.HandshakeInternal(SafeFreeCredentials credential, SafeDeleteSslContext& context, ReadOnlySpan`1 inputBuffer, Byte[]& outputBuffer, SslAuthenticationOptions sslAuthenticationOptions)
   --- End of inner exception stack trace ---
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)
   --- End of inner exception stack trace ---
   at Microsoft.Azure.Devices.Client.Transport.Mqtt.MqttTransportHandler.OpenInternalAsync(CancellationToken cancellationToken)
   at Microsoft.Azure.Devices.Client.Transport.Mqtt.MqttTransportHandler.OpenAsync(CancellationToken cancellationToken)
   at Microsoft.Azure.Devices.Client.Transport.ProtocolRoutingDelegatingHandler.OpenAsync(CancellationToken cancellationToken)
   at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.<>c__DisplayClass27_0.<<ExecuteWithErrorHandlingAsync>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.ExecuteWithErrorHandlingAsync[T](Func`1 asyncOperation)
   at Microsoft.Azure.Devices.Client.Transport.Mqtt.MqttIotHubAdapter.WriteMessageAsync(IChannelHandlerContext context, Object message, Func`3 exceptionHandler)
   at Microsoft.Azure.Devices.Client.Transport.Mqtt.MqttIotHubAdapter.ConnectAsync(IChannelHandlerContext context)
   --- End of inner exception stack trace ---
   at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.ExecuteWithErrorHandlingAsync[T](Func`1 asyncOperation)
   at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.<>c__DisplayClass38_0.<<OpenInternalAsync>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.EnsureOpenedAsync(CancellationToken cancellationToken)
   at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.<>c__DisplayClass23_0.<<EnableMethodsAsync>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.EnableMethodsAsync(CancellationToken cancellationToken)
   at Microsoft.Azure.Devices.Client.InternalClient.SetMethodHandlerAsync(String methodName, MethodCallback methodHandler, Object userContext, CancellationToken cancellationToken)
   at Microsoft.Azure.Devices.Edge.Test.Common.LeafDevice.CreateLeafDeviceAsync(Device device, Func`1 clientFactory, IotHub iotHub, CancellationToken token) in /mnt/vss/_work/1/s/test/Microsoft.Azure.Devices.Edge.Test.Common/LeafDevice.cs:line 321
   at Microsoft.Azure.Devices.Edge.Test.Common.LeafDevice.DeleteIdentityIfFailedAsync(Device device, IotHub iotHub, CancellationToken token, Func`1 what) in /mnt/vss/_work/1/s/test/Microsoft.Azure.Devices.Edge.Test.Common/LeafDevice.cs:line 303
   at Microsoft.Azure.Devices.Edge.Test.Common.LeafDevice.DeleteIdentityIfFailedAsync(Device device, IotHub iotHub, CancellationToken token, Func`1 what) in /mnt/vss/_work/1/s/test/Microsoft.Azure.Devices.Edge.Test.Common/LeafDevice.cs:line 308
   at Microsoft.Azure.Devices.Edge.Test.Common.LeafDevice.CreateWithSasAsync(String leafDeviceId, Option`1 parentId, IotHub iotHub, ITransportSettings transport, String edgeHostname, CancellationToken token, ClientOptions options, Boolean nestedEdge) in /mnt/vss/_work/1/s/test/Microsoft.Azure.Devices.Edge.Test.Common/LeafDevice.cs:line 144
   at Microsoft.Azure.Devices.Edge.Test.Common.LeafDevice.<>c__DisplayClass5_0.<<CreateAsync>b__1>d.MoveNext() in /mnt/vss/_work/1/s/test/Microsoft.Azure.Devices.Edge.Test.Common/LeafDevice.cs:line 62
--- End of stack trace from previous location ---
   at Microsoft.Azure.Devices.Edge.Test.Common.Profiler.Run[T](Func`1 func, String message, Object[] properties) in /mnt/vss/_work/1/s/test/Microsoft.Azure.Devices.Edge.Test.Common/Profiler.cs:line 28
   at Microsoft.Azure.Devices.Edge.Test.AuthorizationPolicy.<>c__DisplayClass0_0.<<AuthorizationPolicyUpdateTest>b__0>d.MoveNext() in /mnt/vss/_work/1/s/test/Microsoft.Azure.Devices.Edge.Test/AuthorizationPolicy.cs:line 93
--- End of stack trace from previous location ---
   at NUnit.Framework.Internal.TaskAwaitAdapter.GenericAdapter`1.BlockUntilCompleted()
   at NUnit.Framework.Internal.MessagePumpStrategy.NoMessagePumpStrategy.WaitForCompletion(AwaitAdapter awaitable)
   at NUnit.Framework.Internal.AsyncToSyncAdapter.Await(Func`1 invoke)

[brycewang-microsoft]

Got agreement from @and-rewsmith that we can close this one.