BackgroundService will not observe exceptions thrown in ExecuteAsync

[Bpoe]

If an exception is thrown in the ExecuteAsync method of a class derived from BackgroundService, the exception is never observed on after StopApplication() is called. This causes the program to exit with Exit Code 0. This causes problems with orchestrators that will not restart a service if it ends with exit code 0.

This is due to this line of code:

runtime/src/libraries/Microsoft.Extensions.Hosting.Abstractions/src/BackgroundService.cs

Line 76 in a21b9a2

await Task.WhenAny(_executeTask, Task.Delay(Timeout.Infinite, cancellationToken)).ConfigureAwait(false);

Task.WhenAny returns a Task<Task>. The parent task will always be RanToCompletion with Exception as null. The Result needs to be awaited or checked for IsFaulted in order for the ExecuteAsync exception to be observed.

[ghost]

Tagging subscribers to this area: @dotnet/area-extensions-hosting
See info in area-owners.md if you want to be subscribed.

Issue Details

---

If an exception is thrown in the ExecuteAsync method of a class derived from BackgroundService, the exception is never observed on after StopApplication() is called. This causes the program to exit with Exit Code 0. This causes problems with orchestrators that will not restart a service if it ends with exit code 0.

This is due to this line of code:

runtime/src/libraries/Microsoft.Extensions.Hosting.Abstractions/src/BackgroundService.cs

Line 76 in a21b9a2

await Task.WhenAny(_executeTask, Task.Delay(Timeout.Infinite, cancellationToken)).ConfigureAwait(false);

Task.WhenAny returns a Task. The parent task will always be RanToCompletion with Exception as null. The Result needs to be awaited or checked for IsFaulted in order for the ExecuteAsync exception to be observed.

Author:	Bpoe
Assignees:	-
Labels:	untriaged, area-Extensions-Hosting
Milestone:	-

[Bpoe]

My quick and dirty fix for my re-implementation of BackgroundService was this:

        public virtual async Task StopAsync(CancellationToken cancellationToken)
        {
            // Stop called without start
            if (this._executeTask == null)
            {
                return;
            }

            try
            {
                // Signal cancellation to the executing method
                this._stoppingCts.Cancel();
            }
            finally
            {
                // Wait until the task completes or the stop token triggers
                var final = await Task.WhenAny(this._executeTask, Task.Delay(Timeout.Infinite, cancellationToken)).ConfigureAwait(false);

                // WhenAny returns a Task<Task>. Its own Task is RanToCompletion without any exceptions, so any exceptions
                // from _executeTask won't be bubbled up.
                // So we can await its result Task that completed and if it had any exceptions, those will now be observed.
                await final;
            }
        }

[eerhardt]

@Bpoe - can you attach a minimal reproduction?

See https://github.com/dotnet/runtime/blob/6732e2ee117e01f29ed81ad293983f9ca602ef37/CONTRIBUTING.md#writing-a-good-bug-report

[Bpoe]

Repro Step:

namespace BackgroundServiceRepro
{
    using System;
    using System.Threading;
    using System.Threading.Tasks;
    using Microsoft.Extensions.DependencyInjection;
    using Microsoft.Extensions.Hosting;

    public class Program
    {
        public static async Task Main(string[] args)
            => await Host
                .CreateDefaultBuilder(args)
                .ConfigureServices((hostContext, services) => services.AddHostedService<Repro>())
                .Build()
                .RunAsync();
    }

    public class Repro : BackgroundService
    {
        private readonly IHostApplicationLifetime hostApplicationLifetime;

        public Repro(IHostApplicationLifetime hostApplicationLifetime)
            => this.hostApplicationLifetime = hostApplicationLifetime;

        protected override async Task ExecuteAsync(CancellationToken stoppingToken)
        {
            await Task.Delay(2000, stoppingToken);

            try
            {
                throw new Exception("Oops!");
            }
            finally
            {
                this.hostApplicationLifetime.StopApplication();
            }
        }
    }
} 

Expected Result

c:\>BackgroundServiceRepro.exe
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Production
info: Microsoft.Hosting.Lifetime[0]
      Content root path: C:\Users\bpoe\source\repos\BackgroundServiceRepro\BackgroundServiceRepro\bin\Debug\net5.0
info: Microsoft.Hosting.Lifetime[0]
      Application is shutting down...
Unhandled exception. System.AggregateException: One or more hosted services failed to stop. (Oops!)
 ---> System.Exception: Oops!
   at BackgroundServiceRepro.Repro.ExecuteAsync(CancellationToken stoppingToken) in C:\Users\bpoe\source\repos\BackgroundServiceRepro\BackgroundServiceRepro\Program.cs:line 32
   at BackgroundServiceRepro.BackgroundService2.StopAsync(CancellationToken cancellationToken) in C:\Users\bpoe\source\repos\BackgroundServiceRepro\BackgroundServiceRepro\Program.cs:line 110
   at Microsoft.Extensions.Hosting.Internal.Host.StopAsync(CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at Microsoft.Extensions.Hosting.Internal.Host.StopAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.WaitForShutdownAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at BackgroundServiceRepro.Program.Main(String[] args) in C:\Users\bpoe\source\repos\BackgroundServiceRepro\BackgroundServiceRepro\Program.cs:line 12
   at BackgroundServiceRepro.Program.<Main>(String[] args)

c:\>echo %errorlevel%
-532462766

Actual Result

c:\>BackgroundServiceRepro.exe
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Production
info: Microsoft.Hosting.Lifetime[0]
      Content root path: C:\Users\bpoe\source\repos\BackgroundServiceRepro\BackgroundServiceRepro\bin\Debug\net5.0
info: Microsoft.Hosting.Lifetime[0]
      Application is shutting down...

c:\>echo %errorlevel%
0

[maryamariyan]

Confirmed repro, and that it is not a regression. Thanks for the report.

[deeprobin]

@maryamariyan I also noticed this behavior recently. I can very gladly take a look (since I am affected myself), if is not a big priority.

If that's okay with you, please assign me so I don't lose track.

[maryamariyan]

@deeprobin how are you affected by this bug?

Have you considered logging when you are calling StopApplication?

Also you could consider using BackgroundServiceExceptionBehavior explained in https://docs.microsoft.com/en-us/dotnet/core/compatibility/core-libraries/6.0/hosting-exception-handling. It stops the host as the default behavior when exception is thrown in background task

Closing as we have no plans to fix this.

[Bpoe]

@maryamariyan - Can you please provide an explanation of why you are not fixing this issue? An also why you closed it as Completed when nothing was done?

The BackgroundServiceExceptionBehavior behavior will cause the application to stop, but it will not observe the exception and the application will exit with error code 0. This prevents orchestrators like Kubernetes, Nomad and Service Fabric from being able to detect a failure in the application and restart it.

[deeprobin]

Let me try to explain it in other words (as I understood the issue of @Bpoe):

In case the BackgroundServiceExceptionBehavior of the host is set to StopHost (as far as I know the default) and an exception occurs an error message is currently logged and the program exits with exit code 0 which means "Successful".

In case of an error, however, it is desired to return an exit code that is not 0.

---

I am not sure if this is a bug or an enhancement. Nevertheless, this is a good point.

[eerhardt]

@Bpoe - In the repro program, why are you calling StopApplication? What is the scenario for throwing an exception, and then call StopApplication()?

Remove that line, and you will see the error logged.

As for the exit code - setting that to non-zero when an exception is thrown is tracked by #67146.

[deeprobin]

Can you please provide an explanation of why you are not fixing this issue?

I could imagine that would be a breaking change.

I mean iirc the host does not terminate the application directly in case of an error but the context is passed to the calling method after termination (in your case probably Program.Main).
You could manually provoke an exit code of course. That would be, however, a (possibly tolerable) Breaking Change.

[deeprobin]

As for the exit code - setting that to non-zero when an exception is thrown is tracked by #67146.

An also why you closed it as Completed when nothing was done?

@eerhardt You could reopen it and close as duplicate (But I think that is not important)