Change feed listener shuts down indefinitely from an InvalidOperationException

[romesh-log]

Over the weekend we encountered an issue where the CosmosDB change feed trigger function stopped processing events entirely.

We saw the following exception in Sentry reporting an issue shutting down the change feed listener, and have seen the issue come up once more since the first time.

Stopping the listener for prefix='xxx', monitoredContainer='PlayerTurnStates', monitoredDatabase='LiveGames', leaseContainer='PlayerTurnStateLeases', leaseDatabase='LiveGames', functionId='xxx.OnPlayerTurnSubmission.Run' failed. Exception: System.InvalidOperationException: Start has to be called before stop
   at Microsoft.Azure.Cosmos.ChangeFeed.FeedManagement.PartitionLoadBalancerCore.StopAsync()
   at Microsoft.Azure.Cosmos.ChangeFeed.FeedManagement.PartitionManagerCore.StopAsync()
   at Microsoft.Azure.Cosmos.ChangeFeed.ChangeFeedProcessorCore.StopAsync()
   at Microsoft.Azure.WebJobs.Extensions.CosmosDB.CosmosDBTriggerListener`1.StopAsync(CancellationToken cancellationToken) in D:\a\_work\1\s\src\WebJobs.Extensions.CosmosDB\Trigger\CosmosDBTriggerListener.cs:line 125.

Expected behaviour

The change feed trigger runs indefinitely or restarts the listener in the event of a failure.

Actual behaviour

The change feed trigger shuts down intermittently and never restarts without manual intervention.

Known workarounds

1. Shut down the function app
2. Delete the lease documents in the leaseContainer
3. Restart the functions app

Related information

Provide any related information

    <PackageReference Include="Microsoft.Azure.Cosmos" Version="3.32.3" />
    <PackageReference Include="Microsoft.Azure.WebJobs.Extensions.CosmosDB" Version="4.2.0" />
    <PackageReference Include="Microsoft.Azure.WebJobs.Extensions.DurableTask" Version="2.9.0" />
    <PackageReference Include="Microsoft.Azure.Functions.Extensions" Version="1.1.0" />
    <PackageReference Include="Microsoft.AspNetCore.Mvc.NewtonsoftJson" Version="3.1.4" />
    <PackageReference Include="Microsoft.Azure.WebJobs.Logging.ApplicationInsights" Version="3.0.35" />
    <PackageReference Include="Microsoft.NET.Sdk.Functions" Version="4.1.1" />

The trigger function in question

[ealsur]

The Listener is controlled by the Functions Runtime and the StartAsync or StopAsync methods are only exposed by the extension, the extension does not control when they get called.

This error means the StopAsync method was called from the runtime while the StartAsync method was still running.

StartAsync creates the host object and then calls Start on it. If StopAsync is called while that is in progress, then this exception will happen. But both these actions are outside of the extension control. This seems to be related to Durable Functions, might be worthing asking those folks on which are the conditions they can be invoked.