[BUG] EventHubConsumerClient.GetPartitionPropertiesAsync throws InvalidOperationException from AMQP library instead of retrying

[tomas-pajurek]

Library name and version

Azure.Messaging.EventHubs 5.11.5

Describe the bug

When calling method GetPartitionPropertiesAsync on type EventHubConsumerClient, exception InvalidOperationException from NuGet Microsoft.Azure.Amqp is thrown with the following message:

"Can't create session when the connection is closing"

I am pretty sure that we are not closing the EventHubConsumerClient or its EventHubConnection prematurely in our code. This claim is supported by:

1. The fact that other concurrently called methods on the same EventHubConsumerClient and other clients using the same EventHubConnection are not throwing at the same time.

2. Historically, there were issues in the Azure's Event Hub and Service Bus SDKs that manifested in the same way:

• Can't create session when the connection is closing #8761
• [BUG] - Microsoft.Azure.ServiceBus - System.InvalidOperationException: Can't create session when the connection is closing. #9416

In both cases, the conclusion was that the thrown InvalidOperationException should be treated properly as an transient exception. Both issues were resolved by catching this exception in the client SDK and wrap it into retriable client exception (which would be EventHubsException in this case). Example of such fix: https://github.com/Azure/azure-sdk-for-net/pull/15984/files

Stack trace:

at Microsoft.Azure.Amqp.AmqpConnection.AddSession(AmqpSession session, Nullable`1 channel)\n  
   at Microsoft.Azure.Amqp.AmqpConnection.CreateSession(AmqpSessionSettings sessionSettings)\n  
   at Azure.Messaging.EventHubs.Amqp.AmqpConnectionScope.CreateManagementLinkAsync(AmqpConnection connection, TimeSpan operationTimeout, TimeSpan linkTimeout, CancellationToken cancellationToken)\n  
   at Azure.Messaging.EventHubs.Amqp.AmqpConnectionScope.OpenManagementLinkAsync(TimeSpan operationTimeout, TimeSpan linkTimeout, CancellationToken cancellationToken)\n  
   at Microsoft.Azure.Amqp.FaultTolerantAmqpObject`1.OnCreateAsync(TimeSpan timeout, CancellationToken cancellationToken)\n  
   at Microsoft.Azure.Amqp.Singleton`1.GetOrCreateAsync(TimeSpan timeout, CancellationToken cancellationToken)\n  
   at Microsoft.Azure.Amqp.Singleton`1.GetOrCreateAsync(TimeSpan timeout, CancellationToken cancellationToken)\n  
   at Azure.Messaging.EventHubs.Amqp.AmqpClient.GetPartitionPropertiesAsync(String partitionId, EventHubsRetryPolicy retryPolicy, CancellationToken cancellationToken)\n  
   at Azure.Messaging.EventHubs.Amqp.AmqpClient.GetPartitionPropertiesAsync(String partitionId, EventHubsRetryPolicy retryPolicy, CancellationToken cancellationToken)\n  
   at Azure.Messaging.EventHubs.EventHubConnection.GetPartitionPropertiesAsync(String partitionId, EventHubsRetryPolicy retryPolicy, CancellationToken cancellationToken)\n  
   at Azure.Messaging.EventHubs.Consumer.EventHubConsumerClient.GetPartitionPropertiesAsync(String partitionId, CancellationToken cancellationToken)\n
   at [redacted]  

Expected behavior

The InvalidOperationException should be wrapped into EventHubsException { IsTransient = true } exception which should be properly retried by existing retry policies within the client SDK.

Actual behavior

The InvalidOperationException is thrown out of the client SDK into the user code unexpectedly.

Reproduction Steps

I not able to deterministically reproduce the error but it happens every few days in our stream processing service running in AKS that periodically calls the problematic GetPartitionPropertiesAsync as well as PartitionReceiver.ReceiveBatchAsync (which never throws such exception).

Environment

• .NET 8.
• Azure Kubernetes Service.
• Event Hub Standard tier.

[github-actions]

Thank you for your feedback. Tagging and routing to the team member best able to assist.

[jsquire]

Hi @tomas-pajurek. Thanks for reaching out and we regret that you're experiencing difficulties. There is special handling in the Azure.Messaging.EventHubs library for this scenario which translates the InvalidOperationException into a transient EventHubsException so that it can be retried. (src)

It looks like we applied that logic to producer and consumer links and did not apply the same pattern to management links. This is confirmed as a bug; we'll get this fixed up.

• Producer (src)
• Consumer (src)
• Management (src)

[jsquire]

fixed by #46544

[jsquire]

The fix has merged and will be included in our November release.

[github-actions]

Hi @tomas-pajurek. Thank you for opening this issue and giving us the opportunity to assist. We believe that this has been addressed. If you feel that further discussion is needed, please add a comment with the text "/unresolve" to remove the "issue-addressed" label and continue the conversation.

[github-actions]

Hi @tomas-pajurek, since you haven’t asked that we /unresolve the issue, we’ll close this out. If you believe further discussion is needed, please add a comment /unresolve to reopen the issue.