FileSystemWatcher is not released properly (OutOfMemoryException)

[Materix]

During integration tests I get OutOfMemoryException for few last tests. For each single test new WebHostBuilder and TestServer were created (so new dependency injection container). After investigation I found that MvcOptions and therefore JsonSerializerSettings are not garbage collected (I have some JsonConverters with reference to quite big configuration).
MvcOptions are hold by PageActionDescrtiptorProvider from Microsoft.AspNetCore.Mvc.RazorPages.Infrastructure

Preparing TestServer and Client

        protected OwnDatabaseServerTestBase()
        {
            var builder = new WebHostBuilder().UseContentRoot(ContentRoot)
                                              .ConfigureAppConfiguration(
                                                  (hostingContext, config) =>
                                                  {
                                                      config.AddJsonFile("config.json", false, true)
                                                            .AddJsonFile($"config.{hostingContext.HostingEnvironment.EnvironmentName}.json", true, true);
                                                  })
                                              .ConfigureServices(InitializeServices)
                                              .UseEnvironment("test")
                                              .UseStartup<TestStartup>();
            TestServer = new TestServer(builder);
            var serviceProvider = TestServer.Host.Services;
            _contextOptions = serviceProvider.GetRequiredService<DbContextOptions<EwidaContext>>();

            using (var context = CreateContext())
            {
                // Required data seeding
            }

            Client = TestServer.CreateClient();

            _availableTokens = new Dictionary<string, (string, string)>();
        }

My test class properly disposed TestServer and HttpClient after each test.

[kichalla]

I think we need to dispose the changetoken that we get here when this singleton service is disposed:
https://github.com/aspnet/Mvc/blob/dev/src/Microsoft.AspNetCore.Mvc.Core/Internal/ActionDescriptorCollectionProvider.cs#L39

[mkArtakMSFT]

@kichalla, assigning this to you to investigate further. Thanks!

[kichalla]

I have some JsonConverters with reference to quite big configuration

Are you holding these converters in a static variable and referencing in each test setup's MvcOptions instance? Also curious what do you mean by big configuration? Could you share how your whole test class(including setup/teardown) looks like?

[Materix]

In Startup (base class for TestStartup) I add converters to DI container and then in implementation of IConfigureOptions<MvcJsonOptions> I add them to MvcJsonOptions (I am not sure if it is good approach to do this, but I need some DI services in my JsonConverters and I solve it by this)

    public static class JsonServiceCollectionExtensions
    {
        public static IServiceCollection AddJson(this IServiceCollection services)
        {
            services.AddOptions();

            // Just one converter for example:
            services.AddSingleton<JsonConverter, TransliteratedStringJsonConverter>();
            // and more....

            services.AddSingleton<NamingStrategy, SnakeCaseNamingStrategy>();
            services.AddSingleton<EwidaContractResolver>();
            services.AddSingleton<IConfigureOptions<MvcJsonOptions>, MvcJsonOptionsConfigurer>();

            return services;
        }
    }

MvcJsonOptionsConfigurer :

    public class MvcJsonOptionsConfigurer : IConfigureOptions<MvcJsonOptions>
    {
        private readonly EwidaContractResolver _contractResolver;
        private readonly IEnumerable<JsonConverter> _converters;

        public MvcJsonOptionsConfigurer(EwidaContractResolver contractResolver, IEnumerable<JsonConverter> converters)
        {
            _contractResolver = contractResolver;
            _converters = converters;
        }

        public void Configure(MvcJsonOptions options)
        {
            options.SerializerSettings.ContractResolver = _contractResolver;
            foreach (var converter in _converters)
            {
                options.SerializerSettings.Converters.Add(converter);
            }
        }
    }

big configuration is IMapper (from AutoMapper) configuration in non-static manner (so for each test new IMapper configuration is created)

[kichalla]

@Materix Could you try doing the following and see if it fixes the issue? Currently the suspect is IHostingEnvironment's implementation which is not a disposable service, so the file providers hanging off of it are not disposed, but this is just a theory and not confirmed yet. I am trying to profile it at my end, but you could try the following meanwhile and see.

Update your server setup like below:

.ConfigureAppConfiguration((webHostBuilderContext, config) =>
                {
                    // Capture the environment created by the builder so that we can dispose the
                    // file providers hanging off of it.
                    HostingEnvironment = webHostBuilderContext.HostingEnvironment;

                    config.AddJsonFile("config.json", false, true)
                              .AddJsonFile($"config.{hostingContext.HostingEnvironment.EnvironmentName}.json", true, true);
                })

And in your test's dispose method, try doing the following:

public void Dispose()
{
    if (HostingEnvironment?.ContentRootFileProvider is PhysicalFileProvider contentPhysicalFileProvider)
    {
        contentPhysicalFileProvider.Dispose();
    }

    if (HostingEnvironment?.WebRootFileProvider is PhysicalFileProvider webRootPhysicalFileProvider)
    {
        webRootPhysicalFileProvider.Dispose();
    }

    TestServer?.Dispose();
}

[Materix]

Yes, this fix the issue. Everything is disposed correctly.

Actually I temporary solved this problem by cleaning out file providers collection for RazorEngine because I don't use it

In Startup.cs:

                    .AddRazorOptions(
                        options =>
                        {
                            options.FileProviders.Clear();
                        }) 

[kichalla]

Awesome. Thanks for the confirmation! Is it possible for you to give a simple repro for this issue. We do run our functional tests quite often and have never come across this kind of issue, so were wondering if you are doing anything different.

[kichalla]

I have created a repro for others to take a look at it:
https://github.com/kichalla/HostingEnvironmentNotDisposedRepro

[Materix]

Actually we have circa 900 integration test, where 700 are based on single static instance of TestServer and remaining 200 are based on instantiating TestServer each time. We started rewriting our tests to server per test mainly because we want to have separate in-memory database for each test and decouple tests from each other (changes in one test were visible in others).

The other root of the problem probably is that our database used in static instance of TestServer grows rapidly (due to bad test arrangement where we use other endpoints and we have audit logic so everything is saved)

I see in your repo that the common scenario is to use single instance of TestServer for all tests. Is it possible to use different database with single TestServer instance (f.e. send name of db in header)? Is there any common practise to achieve this? (we started project on ASP.NET Core 1.0 and updating it)

I uploaded our Startup, TestStartup and both base class for test classes
https://1drv.ms/u/s!ArRvFB3N8tEviU_VsYcDTBFykzB5
(I tried to attach here but github reject it)

[kichalla]

Thanks for sharing the details!

We started rewriting our tests to server per test mainly because we want to have separate in-memory database for each test and decouple tests from each other (changes in one test were visible in others).

👍 That's sounds like a valid scenario for your per test TestServer instance usage.

I see in your repo that the common scenario is to use single instance of TestServer for all tests

We don't use a single instance for all the tests actually. For example, we have the BasicWebSite which is tested by bunch of test classes BasicTests.cs, FiltersTest.cs etc.). In each of these classes we use Xunit's TestFixture way of doing setup of the TestServer only once per that class.

One more usage you might find interesting is you can use different Startup classes for a single web application, so you could have different application setup for the same application and test scenarios differently. For example, BasicWebSite here tests a feature called TempDataProvider by using two different startup types. Note that even though this gives the flexibility of having different startup configuration in different files, it might not help you with the database isolation scenarios you are looking for.

[kichalla]

Closing this issue in favor of https://github.com/aspnet/Home/issues/3110