Add API to help writing integration tests that drive an AppHost project

[DamianEdwards]

For authoring integration tests of web projects, we ship Microsoft.AspNetCore.Mvc.Testing which contains the WebApplicationFactory<TEntryPoint> API. To better support authoring integration tests against an Aspire-based app, we should consider adding something like DistributedApplicationFactory<TEndpointPoint>.

I spiked an approach to this in a branch in the samples repo but it required numerous hacks to make work and doesn't address some other issues that arise when using AppHost projects to orchestrate an Aspire app in a testing context. But it ends up enabling authoring a test like so:

namespace ApiService.Tests;

public class ApiServiceTests : IClassFixture<DistributedApplicationFixture<Program>>
{
    private readonly HttpClient _httpClient;

    public ApiServiceTests(DistributedApplicationFixture<Program> appHostFixture)
    {
        _httpClient = appHostFixture.CreateClient("apiservice");
    }

    [Fact]
    public async void WeatherForecast_Returns_200()
    {
        var forecasts = await _httpClient.GetFromJsonAsync<WeatherForecast[]>("weatherforecast");

        Assert.NotNull(forecasts);
        Assert.NotEmpty(forecasts);
    }

    record WeatherForecast(DateOnly Date, int TemperatureC, string? Summary)
    {
        public int TemperatureF => 32 + (int)(TemperatureC / 0.5556);
    }
}

Note that another approach is to use WebApplicationFactory to host the specific project being tested, and have the test itself create a separate DistributedApplication to manage the target app's dependencies. An example of this approach can be seen in the eShop repo. I think this approach is still useful in some scenarios but results in duplication of the AppHost setup in the test itself, along with needing to manually extract configuration information like connection strings as the app under test is not being setup by the AppHost.

While the example shown above is a good start, the application builder and hosting patterns have evolved since WebApplicationFactory was created, and as such we likely want to enable an updated approach that's more aligned with the modern IHostApplicationBuilder pattern used by WebApplicationBuilder, HostApplicationBuilder, DistributedApplicationBuilder, etc., i.e. a two-phase approach involving a builder and then an app, e.g.:

namespace ApiService.Tests;

public class ApiServiceTests
{
    [Fact]
    public async Task WeatherForecast_Returns_200()
    {
        // arrange
        var builder = DistributedApplicationFactory.CreateBuilder<Program>();
        builder.Services.Add(...); // Services can be added to the DAB here but it's probably uncommon
        builder.Environment = "Test"; // Enable easily setting the environment name, maybe default to "Test"?
        
        await using var app = builder.Build();
        await app.StartAsync(); // Should this start the host *and* wait for DCP startup?

        var client = app.CreateHttpClient("apiservice");

        // act
        var forecasts = await client.GetFromJsonAsync<WeatherForecast[]>("weatherforecast");

        // assert
        Assert.NotNull(forecasts);
        Assert.NotEmpty(forecasts);
    }

    record WeatherForecast(DateOnly Date, int TemperatureC, string? Summary)
    {
        public int TemperatureF => 32 + (int)(TemperatureC / 0.5556);
    }
}

Here's a list of the items we'd likely need to address:

• Update DistributedApplicationBuilder to allow for certain options to be configured via the string[] args passed to the AppHost project's Main method. Today these can only be set in code on the DistributedApplicationOptions instance passed to DistributedApplication.CreateBuilder. Anything proposed below that doesn't currently exist on DistributedApplicationOptions should likely be added there too:
  • The application name, e.g. --application-name. This is used by Aspire.Hosting to find the AppHost assembly upon which a few different things depend, including extracting the path to DCP from the AssemblyMetadata attributes stamped onto the AppHost during build.
  • Disabling the dashboard, e.g. --disable-dashboard.
  • A flag that enables randomization of all ports assigned to resource endpoints, e.g. --randomize-endpoint-ports. This is critical to ensure that concurrent tests against the same resources can be executed without port conflicts.
  • A semi-colon delimited list of resource names to include in the model, e.g. --resource-filter. Resources with names not specified should be removed from the model when built. This enables constraining which resources should run when launching an AppHost. It's the responsibility of the caller to ensure all resources required to satisfy dependencies of the specified resources are included. This is effectively Allow specifying the project(s) on run #303 (rudimentary prototype here).
• Update DistributedApplication such that pre-startup hooks are no longer managed and executed outside of just before starting the inner IHost but rather integrated there such that starting the inner host executes the hooks appropriately.
• Add a new DistributedApplicationFactory.CreateBuilder<TEntryPoint> API to enable configuring and launching an AppHost project from a test. This would be in a new package Aspire.Hosting.Testing. It would use the established HostFactoryResolver pattern to intercept the creation of the internal IHost such that it can be mutated and coordinated as required by the test. Some further details:
  • It should default to passing the args defined above, i.e. --application-name set to the name of the AppHost project assembly of the TEntryPoint type, --disable-dashboard so that the dashboard isn't launched during test execution, --randomize-endpoint-ports so that resource endpoints don't collide with each other when tests are running concurrently (or tests are running while the project itself is still running from a launch session).
  • A method to get an HttpClient instance for a specified resource, and optional endpoint name, e.g. CreateHttpClient(string resourceName, string? endpointName = null). This client have its BaseAddress configured to connect to the appropriate HTTP endpoint for the resource and have standard resiliency added
  • A method to get an allocated endpoint for a specified resource, and optional endpoint name, so that addresses non-http clients can be configured, e.g. GetEndpoint(string resourceName, string? endpointName)
  • A method to get a connection string for a specified resource, e.g. GetConnectionString(string resourceName)
  • A method to build the builder and get back an "app", e.g. Build(). Note that DistributedApplication already implements IAsyncDisposable so this method might return a derived type like WrappedDistributedApplication or perhaps the extra functionality detailed here can be added directly to DistributedApplication or as extensions in Aspire.Hosting.Testing.
  • A method to allow async waiting on the orchestrator to start, rather than just waiting on the IHost itself, e.g. WaitForOrchestrator. Or perhaps the StartAsync method should just default to waiting on the orchestrator instead of the host?
  • A property/method to get the ApplicationModel of the built AppHost project.
  • Console logs from running resources should be collected and piped to the test log to aid in diagnosability.
  • VS test tooling won't attach the debugger to processes launched by DistributedApplication today so we'd likely need to do work there similar to what was done to make launching AppHost projects work well. /Cc @timheuer

[jack775544]

I recently set up some tests against an Aspire application. Some background:

• There are 3 main ASP.NET Core projects in the application
• There is additional ASP.NET Core application running YARP that sits in front of all traffic that is going to the other 3 servers
• There are couple of docker containers that are also running that the main server projects talk to
• Authentication for incoming traffic is done in the YARP proxy and is using EntraID.

I started just trying to use the app host that we use for regular development in the tests but ended up changing to model used in the eShop application with making multiple WebApplicationFactorys and then having a trimmed down distributed app builder for the containers.

The main decisions that made me use this method were:

• Auth - When using EntraID, there is no good way to login to the application in test code. Using the WebApplicationFactory I can modify the ASP.NET host and replace EntraID auth with a test auth service for my testing purposes.
• Invoking individual services - Not all tests that we have are testing API endpoints. Using the WebApplicationFactory we can get an IServiceProvider instance from the server we want to test and then invoke individual services in isolation.

There were also some challenges:

• When making the proxy in the test, we had to make a custom YARP ForwarderHttpClientFactory that would pass requests through to the HttpClient from the correct WebApplicationFactory since the server has been replaced with the in memory test server (and therefore the normal HTTP requests that YARP does won't work).

Personally I would like to see some of merging of the WebApplicationFactory into the Aspire distributed app model for testing. That we can still get fine grained control over the server projects, while having he ease of setup with Aspire.

[mitchdenny]

Assigning to @ReubenBond

[ReubenBond]

I've created a draft PR with my WIP here: #2310

[davidfowl]

Moved to P5

[ReubenBond]

Fixed by #2310

Open new issues for specific follow-ups, eg #2790