Different Exit code when IHost stops from failing (async)IHostedService

[Vandersteen]

Given the following Program.cs

IHost host = Host.CreateDefaultBuilder(args)
    .ConfigureServices(services =>
    {
        services.AddHostedService<Worker>(); 
        services.Configure<HostOptions>(hostOptions =>
        {
            hostOptions.BackgroundServiceExceptionBehavior = BackgroundServiceExceptionBehavior.StopHost;
        });
    })
    .Build();

await host.RunAsync();

And Worker.cs

public class Worker : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        //await Task.Delay(TimeSpan.FromSeconds(1), stoppingToken);
        throw new Exception();
    }
}

I get an 'error' exit code, which is usefull to detect errors in let's say kubernetes

Process finished with exit code 134.

However when uncommenting the 'await Task...'

There is no longer an 'error' exit code

crit: Microsoft.Extensions.Hosting.Internal.Host[10]
      The HostOptions.BackgroundServiceExceptionBehavior is configured to StopHost. A BackgroundService has thrown an unhandled exception, and the IHost instance is stopping. To avoid this behavior, configure this to Ignore; however the BackgroundService will not be restarted.
      System.Exception: Exception of type 'System.Exception' was thrown.
         at TestUpgrade.Worker.ExecuteAsync(CancellationToken stoppingToken) in Worker.cs:line 8
         at Microsoft.Extensions.Hosting.Internal.Host.TryExecuteBackgroundServiceAsync(BackgroundService backgroundService)
info: Microsoft.Hosting.Lifetime[0]
      Application is shutting down...

Process finished with exit code 0. <-------

Is there a way to detect this case using the hostbuilder ?

[ghost]

Tagging subscribers to this area: @dotnet/area-extensions-hosting
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Given the following Program.cs

IHost host = Host.CreateDefaultBuilder(args)
    .ConfigureServices(services =>
    {
        services.AddHostedService<Worker>(); 
        services.Configure<HostOptions>(hostOptions =>
        {
            hostOptions.BackgroundServiceExceptionBehavior = BackgroundServiceExceptionBehavior.StopHost;
        });
    })
    .Build();

await host.RunAsync();

And Worker.cs

public class Worker : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        //await Task.Delay(TimeSpan.FromSeconds(1), stoppingToken);
        throw new Exception();
    }
}

I get an 'error' exit code, which is usefull to detect errors in let's say kubernetes

Process finished with exit code 134.

However when uncommenting the 'await Task...'

There is no longer an 'error' exit code

crit: Microsoft.Extensions.Hosting.Internal.Host[10]
      The HostOptions.BackgroundServiceExceptionBehavior is configured to StopHost. A BackgroundService has thrown an unhandled exception, and the IHost instance is stopping. To avoid this behavior, configure this to Ignore; however the BackgroundService will not be restarted.
      System.Exception: Exception of type 'System.Exception' was thrown.
         at TestUpgrade.Worker.ExecuteAsync(CancellationToken stoppingToken) in /Users/rein/Projects/Skyfresh/system/dotnet/tools/TestUpgrade/Worker.cs:line 8
         at Microsoft.Extensions.Hosting.Internal.Host.TryExecuteBackgroundServiceAsync(BackgroundService backgroundService)
info: Microsoft.Hosting.Lifetime[0]
      Application is shutting down...

Process finished with exit code 0. <-------

Is there a way to detect this case using the hostbuilder ?

Author:	Vandersteen
Assignees:	-
Labels:	untriaged, area-Extensions-Hosting
Milestone:	-

[eerhardt]

The difference in behavior here is that when you don't await in your hosted service, your code runs inline during the process startup. Thus when an exception is thrown before await happens, your startup is failing and the app crashes.

After the await has occurred, the host is started and the background service throws an exception, which the host catches, logs, and then exits the app. In that case the app exists "cleanly" today.

I think it would make sense to set the exit code to something non-zero in this case to indicate that the app has been stopped. Potentially, we may need an option to control the behavior if someone wants to opt-out of it.

Thoughts, @davidfowl @Tratcher @maryamariyan ?

[maryamariyan]

I think it would make sense to set the exit code to something non-zero in this case to indicate that the app has been stopped.

+1. Keeping as-is. Might be good to look for more feedback here before deciding on how to approach this.

[deeprobin]

@maryamariyan In my private kubernetes development cluster I already had also the problem that error handling is hard to show without error code.

Kubernetes dashboard for ex. would otherwise display the status "finished" or something like that instead of "Failed".

But there are many usecases (for ex. Docker)

[davidfowl]

Why not change application code to return a non-zero exit code? This feels more like a template change than a change to these APIs tbh.

[eerhardt]

Why not change application code to return a non-zero exit code?

How does the app know to return a non-zero exit code?

IHost host = Host.CreateDefaultBuilder(args)
    .ConfigureServices(services =>
    {
        services.AddHostedService<Worker>();
    })
    .Build();

await host.RunAsync();

Run and RunAsync return void and Task.

runtime/src/libraries/Microsoft.Extensions.Hosting.Abstractions/src/HostingAbstractionsHostExtensions.cs

Lines 48 to 60 in 8aac5ee

	public static void Run(this IHost host)
	{
	host.RunAsync().GetAwaiter().GetResult();
	}
	/// <summary>
	/// Runs an application and returns a <see cref="Task"/> that only completes when the token is triggered or shutdown is triggered.
	/// The <paramref name="host"/> instance is disposed of after running.
	/// </summary>
	/// <param name="host">The <see cref="IHost"/> to run.</param>
	/// <param name="token">The token to trigger shutdown.</param>
	/// <returns>The <see cref="Task"/> that represents the asynchronous operation.</returns>
	public static async Task RunAsync(this IHost host, CancellationToken token = default)

We would somehow need to bubble the exception up through RunAsync. Or some other way to know that the Host faulted.

[davidfowl]

Then I would change the title of this issue to make sure that RunAsync throws if we're shutting down the host. We can store the exception and make StopAsync throw observe the exception.

[eerhardt]

I think the title is fine - it describes the intended behavior, not the solution.

There are other possible solutions here. One would be to store the caught exceptions on the IHost, and expose them publicly in a property. Then the app code can do whatever they want after RunAsync is done by inspecting the background exceptions.

[deeprobin]

What do you think about simply setting Environment.ExitCode?
(Of course we should not set this behavior as default, because otherwise there might be breaking changes, but this property is the first thing that comes to my mind).

[davidfowl]

We shouldn't be setting the exit code. We used to do that a long time ago in the library because of other issues but the application owns the entrypoint.

There are other possible solutions here. One would be to store the caught exceptions on the IHost, and expose them publicly in a property. Then the app code can do whatever they want after RunAsync is done by inspecting the background exceptions.

Maybe it's both? RunAsync is one observer of the Exception that was captured, which rethrows.

[tmds]

Exiting due to failure with a success exit code is bad.

We can store the exception and make StopAsync throw observe the exception.

This seems like the proper thing to do. StopAsync already throws an AggregateException when IHostedServices throw during StopAsync.

We can't capture the exception for the generic IHostedServices because they don't have a Task that tracks the execution.
It is possible for BackgroundService, which has such a Task (ExecuteTask property).
(note: ExecuteTask isn't set if someone overrides StartAsync but forgets to call the base version.)

[deeprobin]

We can't capture the exception for the generic IHostedServices because they don't have a Task that tracks the execution.

@tmds
How about try-catch?

[Dean-NC]

I agree with the ability to have a different exit code. I'd like my Windows service's recovery to get done (restart), but with BackgroundServiceExceptionBehavior.StopHost I can't tell if the host is stopping due to an unhandled error. IHostApplicationLifetime.ApplicationStopping doesn't help...nothing to check in there.

[msschl]

I think an other BackgroundServiceExceptionBehavior value on the enum which sets an exit code on an unhandled error would be good. For Kubernetes when you use a dotnet worker as a cronjob the job is shown as completed with no failure although the background service might have failed.

[DamianEdwards]

Extension I put together just now for a template we're working on that ensures the app exits with a non-zero exit code if a BackgroundService faults:

public static class HostExtensions
{
    public static async Task RunAndObserveBackgroundServices(this IHost host)
    {
        var backgroundServiceTasks = host.Services.GetServices<IHostedService>()
            .OfType<BackgroundService>().Select(s => s.ExecuteTask);

        await host.RunAsync();

        if (backgroundServiceTasks.Any(t => t?.IsFaulted == true))
        {
            // If a BackgroundService failed, set the exit code to -1.
            Environment.ExitCode = -1;
        }
    }
}

Simply call this method instead of RunAsync()/Run() from your Program.Main.

Alternative is add an additional IHostedService like the folllowing:

class BackgroundServiceObserver(IServiceProvider serviceProvider) : IHostedService
{
    public Task StartAsync(CancellationToken cancellationToken) => Task.CompletedTask;

    public Task StopAsync(CancellationToken cancellationToken)
    {
        var backgroundServiceTasks = serviceProvider.GetServices<IHostedService>()
            .OfType<BackgroundService>().Select(s => s.ExecuteTask);

        if (backgroundServiceTasks.Any(t => t?.IsFaulted == true))
        {
            // If a BackgroundService failed, set the exit code to -1.
            Environment.ExitCode = -1;
        }

        return Task.CompletedTask;
    }
}

Given the impact of the current behavior in environments like Kubernetes, Azure Container Apps, Aspire, etc., I think we should prioritize improving this behavior in the runtime. As @msschl suggest, we should add a new value to BackgroundServiceExceptionBehavior (e.g. StopHostAndSetExitNonZeroCode, aka my best name suggestion ever) in .NET 10. We should also consider servicing .NET 8 and .NET 9 to enable this behavior via an app context switch (i.e. quirk it) so that folks can get the benefit of this change without having to wait for .NET 10.