System.OverflowException: Arithmetic operation resulted in an overflow at Google.Protobuf.MessageExtensions.ToByteArray(IMessage message)

[richardpaynea55]

Description

I am using Durable functions with several activities - several in parallel. The functions have worked previously, both locally and in production. We have had several failures, however, with the same error: Function 'Orchestrator (Orchestrator)' failed with an error. Reason: Microsoft.Azure.WebJobs.Host.FunctionInvocationException: Exception while executing function: Functions.Orchestrator

Expected behavior

Expected function to complete successfully.

Actual behavior

We have observed an Orchestration Failure exception being thrown each time the function runs. In code all activities are invoked within a try - catch block, but the Failed exception does not get caught there.

Function 'Orchestrator (Orchestrator)' failed with an error. Reason: Microsoft.Azure.WebJobs.Host.FunctionInvocationException: Exception while executing function: Functions.Orchestrator
---> Microsoft.Azure.WebJobs.Script.Workers.Rpc.RpcException: Result: Failure
Exception: System.OverflowException: Arithmetic operation resulted in an overflow.
at Google.Protobuf.MessageExtensions.ToByteArray(IMessage message)
at Microsoft.DurableTask.Worker.Grpc.GrpcOrchestrationRunner.LoadAndRun(String encodedOrchestratorRequest, ITaskOrchestrator implementation, IServiceProvider services)
at Microsoft.Azure.Functions.Worker.Extensions.DurableTask.DurableTaskFunctionsMiddleware.RunOrchestrationAsync(FunctionContext context, BindingMetadata triggerBinding, FunctionExecutionDelegate next) in /_/src/Worker.Extensions.DurableTask/DurableTaskFunctionsMiddleware.cs:line 58
at Microsoft.Azure.Functions.Worker.FunctionsApplication.InvokeFunctionAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\FunctionsApplication.cs:line 89
at Microsoft.Azure.Functions.Worker.Handlers.InvocationHandler.InvokeAsync(InvocationRequest request) in D:\a\_work\1\s\src\DotNetWorker.Grpc\Handlers\InvocationHandler.cs:line 88
Stack: at Google.Protobuf.MessageExtensions.ToByteArray(IMessage message)
at Microsoft.DurableTask.Worker.Grpc.GrpcOrchestrationRunner.LoadAndRun(String encodedOrchestratorRequest, ITaskOrchestrator implementation, IServiceProvider services)
at Microsoft.Azure.Functions.Worker.Extensions.DurableTask.DurableTaskFunctionsMiddleware.RunOrchestrationAsync(FunctionContext context, BindingMetadata triggerBinding, FunctionExecutionDelegate next) in /_/src/Worker.Extensions.DurableTask/DurableTaskFunctionsMiddleware.cs:line 58
at Microsoft.Azure.Functions.Worker.FunctionsApplication.InvokeFunctionAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\FunctionsApplication.cs:line 89
at Microsoft.Azure.Functions.Worker.Handlers.InvocationHandler.InvokeAsync(InvocationRequest request) in D:\a\_work\1\s\src\DotNetWorker.Grpc\Handlers\InvocationHandler.cs:line 88
at Microsoft.Azure.WebJobs.Script.Description.WorkerFunctionInvoker.InvokeCore(Object[] parameters, FunctionInvocationContext context) in /_/src/WebJobs.Script/Description/Workers/WorkerFunctionInvoker.cs:line 101
at Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase.Invoke(Object[] parameters) in /_/src/WebJobs.Script/Description/FunctionInvokerBase.cs:line 82
at Microsoft.Azure.WebJobs.Script.Description.FunctionGenerator.Coerce[T](Task`1 src) in /_/src/WebJobs.Script/Description/FunctionGenerator.cs:line 225
at Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`2.InvokeAsync(Object instance, Object[] arguments) in D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionInvoker.cs:line 53
at Microsoft.Azure.WebJobs.Extensions.DurableTask.OutOfProcMiddleware.<>c__DisplayClass10_0.<<CallOrchestratorAsync>b__0>d.MoveNext() in D:\a\_work\1\s\src\WebJobs.Extensions.DurableTask\OutOfProcMiddleware.cs:line 130
--- End of stack trace from previous location ---
at Microsoft.Azure.WebJobs.Host.Executors.TriggeredFunctionExecutor`1.<>c__DisplayClass7_0.<<TryExecuteAsync>b__0>d.MoveNext() in D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\TriggeredFunctionExecutor.cs:line 51
--- End of stack trace from previous location ---
at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.InvokeWithTimeoutAsync(IFunctionInvoker invoker, ParameterHelper parameterHelper, CancellationTokenSource timeoutTokenSource, CancellationTokenSource functionCancellationTokenSource, Boolean throwOnTimeout, TimeSpan timerInterval, IFunctionInstance instance) in D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 581
at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithWatchersAsync(IFunctionInstanceEx instance, ParameterHelper parameterHelper, ILogger logger, CancellationTokenSource functionCancellationTokenSource) in D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 523
at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstanceEx instance, FunctionStartedMessage message, FunctionInstanceLogEntry instanceLogEntry, ParameterHelper parameterHelper, ILogger logger, CancellationToken cancellationToken) in D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 306
--- End of inner exception stack trace ---
at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstanceEx instance, FunctionStartedMessage message, FunctionInstanceLogEntry instanceLogEntry, ParameterHelper parameterHelper, ILogger logger, CancellationToken cancellationToken) in D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 352
at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.TryExecuteAsync(IFunctionInstance functionInstance, CancellationToken cancellationToken) in D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 108. IsReplay: False. State: Failed. RuntimeStatus: Failed. HubName: <Name>. AppName: <Name>. SlotName: <SlotName>. ExtensionVersion: 2.13.4. SequenceNumber: 143. TaskEventId: -1

Relevant source code snippets

The Orchestration function takes an input message and then goes through a set of activities - one part is performed in parallel:

The code is as follows:

var message = context.GetInput<Message>();

if (!context.IsReplaying)
{
	log.LogInformation($"Message obtained for Upload {message.Id}.");
}

var fileName = string.Empty;
var cache = string.Empty;
var azureStorageTableName = string.Empty;

try
{
	// 1. Initialise and get cache for processing
	cache = await context.CallActivityAsync<string>("Initialise", message);

	azureStorageTableName = await context.CallActivityAsync<string>("CreateAzureDataTable", message);

	// 2. Initialise data in table
	var fileProps = await context.CallActivityAsync<HandleFileActivity.HandleFileResult>(
		"HandleFileAsync",
		new HandleFileActivity.HandleFileRequest(azureStorageTableName, cache));

	// 3. Get the batches
	var batches = await context.CallActivityAsync<List<BatcherActivity.BatchResult>>(
		"GetBatches",
		new BatcherActivity.BatchRequest(fileProps));

	var parallelTasks = new List<Task<string>>();

	foreach (var batch in batches)
	{
		// 4. Set up tasks to run in parallel - handling a batch per process.
		var task = context.CallActivityAsync<string>(
			"ProcessBatch",
			new BatchActivity.ProcessBatchRequest(batch, cache, azureStorageTableName));

		parallelTasks.Add(task);
	}

	// 5. Wait for all results to come back
	await Task.WhenAll(parallelTasks);

	// 6. Aggregate the batch statistics results
	var batchResults = new List<string>();
	parallelTasks.ForEach(task => batchResults.Add(task.Result));

	// 7. Complete the process
	await context.CallActivityAsync(
		"Completion",
		new CompletionActivity.CompletionRequest(batchResults));

	await context.CallActivityAsync("Cleanup", azureStorageTableName);
}
catch (Exception e)
{
	var reason = "An unexpected error has occurred.";
	log.LogError(reason, e);

	await context.CallActivityAsync("Failed", new FailureActivity.FailureRequest(cache, reason, fileName));

	await context.CallActivityAsync("Cleanup", azureStorageTableName);
}

Known workarounds

No known workarounds.

App Details

• Durable Functions extension version (e.g. v1.8.3): v1.1.5
• Azure Functions runtime version (1.0 or 2.0): 2.0
• Programming language used: C#

If deployed to Azure

• Timeframe issue observed: 13/01/2028 07:28 - 08:28 UTC
• Orchestration instance ID(s):
  • 7749d1f6-a761-4cd1-b517-d0992c77bb0e,
  • 57eaf6cc-ea21-45ce-8d17-390c52aefbf5,
  • 5ab63f0a-e52f-4839-8959-14e9c1db7ffb,
  • e1dc203c-a7d1-4c1a-bef5-298330ecbb40,
  • 2aa14bc2-78de-4457-a0cd-587b2d84d549,
  • 46d83274-7fc6-4388-ab5e-9516f8deff29.

[richardpaynea55]

To add to this - I know the Function has started running and performing at least two of the activities.

The HandleFileActivity above:

// 2. Initialise data in table
	var fileProps = await context.CallActivityAsync<HandleFileActivity.HandleFileResult>(
		"HandleFileAsync",
		new HandleFileActivity.HandleFileRequest(azureStorageTableName, cache));

Copies data into an Azure Storage table - I can see that that data is present.