Blazor Server Memory Leak

[ConnorJamesLow]

Describe the bug

Old data isn't cleared from memory, regardless of time passage and CircutOptions configuration.

To Reproduce

Project

Repository Link

I've experienced this in a production app, but I have recreated the behavior using the default blazor server template. I made the following changes:

Startup.cs - reduce retained connections.

services.AddServerSideBlazor(o => {
    o.DisconnectedCircuitMaxRetained = 1;
    o.DisconnectedCircuitRetentionPeriod = TimeSpan.FromSeconds(10);
});

Data/WeatherForcastService.cs - Add variable forecast length.

public class WeatherForecastService
{
    // ...

    public Task<WeatherForecast[]> GetForecastAsync(DateTime startDate, int count) // added `count` parameter
    {
        var rng = new Random();
        return Task.FromResult(Enumerable.Range(1, count).Select(index => new WeatherForecast
        { // ...

Pages/FetchData.razor - add controls for appending x amount of forecast rows.

else
{
    <span>Load</span>
    <input type="number" @bind="Count"/>
    <span>Items</span>
    <button @onclick="LoadMore">Go</button>

@* ... *@

@code {
    private List<WeatherForecast> Forecasts { get; set; } = new();
    private int Count { get; set; } = 5;
    private async Task LoadMore() => Forecasts
        .AddRange(await ForecastService.GetForecastAsync(DateTime.Now, Count));

    // ...
}

Steps

1. Navigate to /fetchdata
2. Load a large number of rows (e.g. 10,000)
3. Navigate away from the page
4. Repeat 1-3 until memory grows substantially (should jump ~ 25 MB / 10,000 rows).

Memory is never released at this point, regardless of time passed or page closed/reloaded. Even if it was, why does memory grow this way? I would expect it to release memory when I navigate away because that data is never accessed after that point.

Exceptions (if any)

n/a

Further technical details

ASP.NET Core version: net5.0

Include the output of dotnet --info:

.NET SDK (reflecting any global.json):
 Version:   6.0.100-preview.3.21202.5
 Commit:    aee38a6dd4

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.19043
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\6.0.100-preview.3.21202.5\

Host (useful for support):
  Version: 6.0.0-preview.3.21201.4
  Commit:  236cb21e3c

.NET SDKs installed:
  5.0.100 [C:\Program Files\dotnet\sdk]
  5.0.102 [C:\Program Files\dotnet\sdk]
  5.0.202 [C:\Program Files\dotnet\sdk]
  5.0.300-preview.21180.15 [C:\Program Files\dotnet\sdk]
  6.0.100-preview.3.21202.5 [C:\Program Files\dotnet\sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.All 2.1.26 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.27 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.26 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.27 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.13 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.14 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 6.0.0-preview.3.21201.13 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.1.26 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.27 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.13 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.14 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.0-preview.3.21201.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 3.1.13 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 3.1.14 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 5.0.0 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 5.0.4 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 5.0.5 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 6.0.0-preview.3.21201.3 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

The IDE (VS / VS Code/ VS4Mac) you're running on, and its version: Microsoft Visual Studio Community 2019, Version 16.9.4

[ConnorJamesLow]

Per #30210 suggestions, I added a GC.Collect triggered by a button. After closing a window and waiting about 20 seconds, I opened the app again and triggered GC. Task Manager reported about 50% of the memory freed. Still, that leaves quite a bit of unused memory, and I can't be triggering GC in production like that: am I missing some configuration? This behavior is not desirable as it puts a lot of strain on our production server.

[davidfowl]

The garbage collector triggers collections when you allocate. What behavior are you looking for? You can configure GC limits https://docs.microsoft.com/en-us/dotnet/core/run-time-config/garbage-collector

You might also consider taking a GC trace and sharing it (dotnet trace collect --profile gc-collect)

[SteveSandersonMS]

Also as per another comment in #30210:

Blazor keeps a limited number of circuits in memory for up to 3 minutes (by default) to enable applications to reconnect to the same session in the event that there is an intermittent network connection.

... you would need to wait 3 minutes before triggering the collection in order for an abandoned circuit's data to be released. Please see DisconnectedCircuitRetentionPeriod if you want to change the config for this.

[ConnorJamesLow]

^{...There should really be an undo button for that.}
Steve, please see the original issue I raised. I explained that I configured this behavior in Startup.cs:

services.AddServerSideBlazor(o => {
    o.DisconnectedCircuitMaxRetained = 1;
    o.DisconnectedCircuitRetentionPeriod = TimeSpan.FromSeconds(10);
});

[SteveSandersonMS]

Thanks for clarifying. I didn't spot that within the collapsed code blocks.

I'd say the main two ways for trying to diagnose what's going on would be:

1. Collecting a GC trace as per @davidfowl's suggestion
2. Trying to see if you can create an out-of-memory condition by repeatedly creating many connections then closing their tabs, or by performing some action repeatedly as a single user. If your repro code will actually crash with an out-of-memory error, that does suggest a leak. Whereas if the memory only grows up to a particular point then it's probably normal GC behavior.

[ConnorJamesLow]

I took a (dotnet) trace recording events incurring memory and the forced GC.Collect after a 10+ second wait:
ScrollingMemoryGrowth.exe_20210512_082522.zip

[ConnorJamesLow]

I tried what Steve suggested and attempted to create an out-of-memory condition. The app consistently gc'd 1-2 GB when the memory total reached ~3.5GB, so no memory leak.

I understand that this is a stateful application and will consume more memory than a stateless web API, for example. I am wondering if there is any reason GC can't occur more often? With enough time, I can get our production app to consume 2+ GB of memory for a single user under normal usage, which does not seem very scalable. Unlike the example app, we are putting limits on list lengths, pagination, etc. Is this something I can configure in the GC limits?

[davidfowl]

The GC doesn't work that way, it doesn't run on a timer. It runs based on what you allocate. When you have memory hanging around but no allocations it won't do anything until one of the thresholds are reached, not even with enough time.

I can get our production app to consume 2+ GB of memory for a single user under normal usage, which does not seem very scalable

It might make more sense to use client GC rather than server GC here. It should scale fine but it is pretty lazy to collect. For these scenarios though, that may not make as much sense.

Does you application have lots of concurrency?

[ConnorJamesLow]

Is client GC the same thing as Workstation GC?

Yes, our application has a fair amount of concurrency.

[davidfowl]

Is client GC the same thing as Workstation GC?

Yes

Yes, our application has a fair amount of concurrency.

What's fair amount? 1000, 10000 concurrent users?

What does your hardware look like? RAM?